Method and system for improving utilization of human searchers

ABSTRACT

A method and system for improving utilization of human searcher(s) in a search system including utilizing a processing such as Natural Language Processing to improve the ability of the search system to recognize an intent of a user request, provide information to an expeditor to respond to queries, and automatically route a query to a relevant and qualified searcher. Using NLP, a request is processed to allow improved use of automated responses, more effective use of the expeditor, and more effective use of search specialists.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to and claims the benefit of U.S.application Ser. No. 12/275,864, Attorney Docket No. 1918.1034, inventorScott A. Jones, et al., titled METHOD AND SYSTEM FOR IMPROVINGUTILIZATION OF HUMAN SEARCHERS, filed Nov. 21, 2008, U.S. applicationSer. No. 12/434,385, Attorney Docket No. 1918.1036, inventor Scott A.Jones, et al., titled METHOD AND SYSTEM FOR IMPROVEMENT OF REQUESTPROCESSING, filed May 1, 2009, and U.S. Provisional Application Ser. No.61/119,205, Attorney Docket No. 1918.1064, inventor Mike Burroughs, etal., titled METHOD AND SYSTEM FOR IMPROVING UTILIZATION OF HUMANSEARCHERS, filed Dec. 2, 2008 the disclosures of which are incorporatedherein by reference in their entirety.

BACKGROUND

1. Field

Embodiment(s) described herein relate to search engine technologiesincluding human-assisted search engines and, more particularly, to amethod and system of processing a request for information using acombination of automated processing and human intelligence.

2. Description of the Related Art

A human assisted search service can provide a more natural and effectiveresponse to a request for information. Natural language queries may beinterpreted, and a response to a query may be obtained by a person. Thismay be particularly desirable if a query is submitted using a mobiledevice which may have limited capabilities. In particular, the use ofSMS or voice to submit a request for information may be more readilyaccepted when a human-assisted response is provided.

Human assisted search services have found greater acceptance than purelyautomated services due at least in part to a higher answer quality,increased ease of use, and generalized response capability. Howeverhuman-assisted search systems must generally have a low cost to serve auser in order to find broad acceptance. For example, pay-per-use systemssuch as any question answered (AQA), or lnfonxx Knowledge GenerationBureau have answered a few million queries per year, while free servicessuch as ChaCha® have answered tens of millions of queries per month.

Regardless of whether a human assisted search system is pay-per-use oradvertising supported, the cost to provide a response to a query is adirect impact to profitability. However, the ability to reduce cost isquite difficult. Comparison of a user request to a database of previousrequests may be ineffective even if the database of previous queries islarge. Factors including such as the combinatorial nature of humanlanguages, spelling, slang or colloquialisms, etc. may impair theability to match a query to a stored query. If a query is passed to aperson, it may be difficult for the person to select an answer withouthaving knowledge of the process applied to the query.

In light of the above and other problems existing in typical searchsystems, there is a need for a system and method for optimizing use ofhuman searchers utilizing natural language processing.

SUMMARY

A method and system of improving utilization of human searcher(s)including using Natural Language Processing to recognize an intent of auser request, provide information to an expeditor to respond to queries,and automatically route a query to a relevant searcher.

A computer-implemented method is provided herein that executes anoperation including receiving a request for information, executing aprocessing with respect to the request including comparing a predefinedquery pattern, correcting spelling, substituting words, removingsignatures using information of stored requests, creating automaticallyan annotation graph for the request, selecting a category, obtainingqueries determined to match the annotation graph using an index ofqueries, calculating a matching function comparing the queries to theannotation graph, calculating a rating of a search result associatedwith the queries, ranking the queries based on the matching function,ranking the search results based on the rating and providing a highestranked search result responsive to the request.

A system embodiment, methods, data structures and interfaces foroperating the embodiments are disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and advantages of the invention will become apparent and morereadily appreciated from the following description of the embodiments,taken in conjunction with the accompanying drawings, of which:

FIG. 1 is a block diagram of a system embodiment.

FIG. 2 illustrates a flow diagram of a process of performing a search.

FIG. 3 illustrates a flow diagram of a process of responding to arequest.

FIG. 4 illustrates a flow diagram of a process of responding to a searchresult.

FIG. 5 illustrates a database record for a raw query.

FIG. 6 illustrates a database record for a vetted query.

FIG. 7 illustrates a flow diagram of a process of performing a search.

FIG. 8 illustrates a flow diagram of a process of processing a searchrequest.

FIG. 9 illustrates a flow diagram of a process of reusing a searchresult.

FIG. 10 illustrates a flow diagram of a process of routing a searchrequest.

FIG. 11 illustrates a flow diagram for vetting a request.

FIG. 12 illustrates a query processing GUI for preparing a search for asearcher.

FIG. 13 illustrates a query processing GUI for selecting a searchresult.

FIG. 14 illustrates a GUI for selecting a search result provided by aresource.

FIG. 15 illustrates a GUI for selecting a search result provided by aresource.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the present embodimentsdiscussed herein, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below to explain the disclosedsystem and method by referring to the figures. It will nevertheless beunderstood that no limitation of the scope is thereby intended, suchalterations and further modifications in the illustrated device, andsuch further applications of the principles as illustrated therein beingcontemplated as would normally occur to one skilled in the art to whichthe embodiments relate. As used herein words importing the singularshall include the plural and vice versa unless specifically counterindicated.

A method and system for improving utilization of human searcher(s) usingnatural language processing (NLP) includes using NLP to process aninitial query. Should the NLP be unable to determine an intent of acustomer, a system may utilize information from the NLP process todetermine if a request may be handled by a human assistant specializingin query processing or ‘expediter’ or may be routed directly to a searchspecialist or ‘searcher’ to perform a search. NLP results may be used toassist an expediter in using a toolset which may enable a rapid andrepeatable response to various user requests. Subsequent to processingby an expediter, NLP may be applied to a request in order to determineif a previous answer may be provided and/or if a searcher should obtaina search result.

Processing such as algorithmic processing of a request is applied inorder to optimize the use of human guides. A request is first comparedto standard templates to determine if the request conforms to anautomated response format. If a query is not found to conform to arecognized format, the query is analyzed using NLP. NLP may be used toremove information from and/or augment a query such that the query maymore easily be categorized and/or associated with a recognized format.

Subsequent to the NLP process, a request is compared to a database ofrequests which may be used to determine whether a search result isavailable which may be used to provide a response to the request. If itis determined that a search result is not available, the results of NLPmay be used to determine if an ambassador guide or a searcher shouldreceive the request.

A method and system of improving utilization of human searcher(s) isdisclosed. A query is received by a search system, which may apply NLPto the query. If a query is received, it may be processed to determineif the query is a formulaic or “keyword query”. A “keyword query” is aquery which begins with and/or includes a word and/or phrase whichindicates that the query is intended for automated processing. Forexample, a query which begins with the word “weather” and includes anoun phrase which is recognized as a location may be determined to be akeyword query. A request may be determined to be a “keyword query” basedon one or more factors including when the request contains at least oneword pre-identified by a user (who may be an administrator) as being acause for an automated processing, based on previous association of thequery with automated processing, etc.

NLP of a query may be applied to make the query easier to interpretand/or to remove extraneous information. For example, spellingcorrection, grammar correction, word substitution, and signature removalmay be applied to improve the probability that a query may be recognizedautomatically and/or using a human assistant. NLP may include namedentity analysis. A ‘named entity’ may be any noun or noun phrase whichis associated with a proper noun. For example, a database ofcelebrities, sports figures, cities, neighborhoods, etc. may be used toobtain a lexicon for named entity analysis.

NLP of a query may include categorization and reformatting of a query.For example, a Bayesian model for categorization may be applied, basedon factors such as noun phrases associated with a query which may beassociated with an index of categories. In at least one embodiment,grammar of a query may be examined to apply parts of speech tags to thequery.

After a query has been processed using NLP, the query may be compared totemplates which may allow a query to be answered automatically. If ananswer can not be provided programmatically (or automatically), aprocessed query may be compared to a database of stored queries, and ifit is determined that a query is a match to a stored query, an answerassociated with the stored query may be provided responsive to thequery. While a query is being described as being compared to a template,the present invention is not limited thereto. For example, a query maybe compared to one or more formatted, predefined or standardized queriesincluding to determine whether the query can be automatically answeredusing one or more of the responses of matching quer(ies).

A query may be analyzed to determine whether an expediter is to bepresented with the query. If an expediter is presented with the query, aprocessed query and a user query maybe presented to the expediter. Anexpediter may reformat a query, associate a location with a query and/ormay categorize a query, and a resulting query may be processed to obtainqueries which may match the resulting (revised or reformatted) query. Ifan expediter has modified and/or augmented a query, parameters forselection of an answer and/or matching of a query to a stored query maybe modified. An expediter may override a system matching.

If a query is directed to a searcher, the searcher may receiveinformation of an unprocessed, a NPL processed, and an expediterprocessed forms of a query. A categorization associated with a query mayindicate whether an expediter has selected the categorization.

A system embodiment, methods, data structures and interfaces foroperating the embodiments are described below in detail.

A “user” may be any person or entity which may submit a request orsearch request. A request or search request or query is informationwhich may be submitted using any communication service which may be ableto access a search system as further described herein below. A query mayinclude text, a word, keywords, and/or a fully formed question. A querymay include any media which are compatible with an interface to a searchservice such as text, images, audio, video, URL's, etc.

A “guide” may be any person who may be compensated and/or may be avolunteer who may respond to a request. An “ambassador” or “expediter”is a guide who may perform processing of a request and/or a searchresult(s). A “searcher” is a guide who may perform an information searchresponsive to a request.

A “raw query” is a request submitted by a user, which may include anytype of information provided by a user and/or associated with a user.

A “vetted query” includes a request which is associated with a category,a structured query, or otherwise qualified query.

A “structured query” is a question formulated according to a structuredgrammar. In at least one embodiment, a grammar construction required fora structured query is a question. A structured query may also bereferred to as a “succinct query”.

An “identifier” or ID is a group of characters, numbers and/or othertypes of information which may be used to identify an item.

A “guided request” is a request which uses the assistance of one or moreguides.

A “result” or “search result” is any information which may be providedresponsive to a request. A search result includes but is not limited toany of an advertisement(s), a link to a web page, a message of any sort,image, audio, text, games, interactive media and/or software of anysort.

A “search resource” is any source of information which may be used toobtain a search result. A search resource includes automated and/orhuman-assisted systems, and any type of media and/or systems which mayprovide information. For example, web pages, search engines, data feeds,databases, a private archive, printed materials, etc. may be a searchresource.

A “profile” is a group of one or more characteristics which may beassociated with a person. Profile characteristics include but are notlimited to demographic, geographic, personality, affiliations, areas ofinterest, historical actions, preferences, memberships, associations,etc.

In order to improve utilization of searchers, automation may beutilized. However, automated processing may degrade system performanceas algorithmic analysis may not meet a needed level of confidence.However, a blended approach to use of human intelligence and algorithmicprocessing may provide a high quality user experience while making ahuman assisted system cost effective. In a human assisted system,Natural Language Processing (NLP) may be used in order to improve thecontent of a query prior to presenting the query to a human assistant orguide.

NLP may be used to correct errors, detect the language of a query, andto make other corrections including as is well established in the art.In the new system NLP is further used to estimate a match between a userrequest and a structured query. NLP is used to evaluate a quality ofstored search results associated with a query in order to determinewhether a stored result may be provided to a user. NLP is further usedto optimize the use of ambassadors. If NLP determines a most probabletype of query, an ambassador may be provided with tools associated withthe most probable type of response. Likewise if a request is categorizedwith a high level of confidence and a structured query has beendetermined to match a user request with high probability, a request maybe provided to a searcher without utilizing an ambassador.

An exemplary system embodiment is illustrated in FIG. 1. As illustratedin FIG. 1, system 100 includes searcher systems 105, 110, a network 115such as the Internet, a search system 130, user systems 135, 140, adatabase 120, which may comprise various records, and ambassador system145, 150.

Each of the guide systems, that is, the searcher systems 105, 110 andthe ambassador systems 145 and 150 can be operated by a guide to obtaina search result on behalf of a user located at a user system (e.g., theuser systems 135, 140). While only a limited number of searcher (alsoreferred to as a human searcher), ambassador (also referred to as avetting guide or query vetting guide or expediter) and user (alsoreferred to as an information seeker or requester) systems are depictedin FIG. 1, it is within the scope of the disclosure for multiplesearcher, ambassador and user systems to be utilized.

The network 115 (FIG. 1) may be a global public network of networks (theInternet) and/or consist in whole or in part of one or more privatenetworks and communicatively couples the searcher computer systems 105,110, the ambassador systems 145, 150 and user computer systems 135, 140with the other components of the system 100 such as the search system130, and the database 120. The network 115 may for example include anyor all of a wireless network(s) such as a cellular phone network, aWiMax network, a WiFi network, etc. a network such as the PublicSwitched Telephone Network (PSTN), and/or any other type ofcommunication network.

The search system 130 allows interaction to occur between the searchersystems 105, 110, the ambassador systems 145, 150 and the user systems135, 140. For example, a request can be transmitted from the usersystems 135, 140 to the search system 130, where a request can beaccessed by the searcher systems 105, 110 and/or the ambassador systems145, 150. Similarly, a search result produced using the searcher systems105, 110 or the ambassador systems 145, 150 in response to a requestproduced by the user systems 135, 140 may be transmitted to the searchsystem 130, where it may be stored by the search system 130 and/or maybe transmitted to the user systems 135, 140 and/or the ambassadorsystems 145, 150.

The user systems 135, 140 may include within the scope of thedisclosure, any device through which an information seeker can submit arequest to and/or receive information or search result from the searchsystem 130. In one embodiment, the user system 135 may be a deviceconfigured for connection to a network and may run web browser software.Thus, the user systems 135, 140 may be a portable or laptop computer, apersonal data assistant (PDA), a desktop computer such as a PC or Mac®,a workstation or a terminal connected to a mainframe, a specializedsearch apparatus, or a smart phone, etc. within the scope of thedisclosure. Any device which can communicate using Internet Protocol(IP) may be a user system, a searcher system, and/or an ambassadorsystem within the scope of the disclosure.

The search system 130 may include a gateway for voice communication anda speech-to-text system or other transcription device and/or personnelto facilitate access to the search system via voice communications suchas through a land line phone, cellular phone, Voice over InternetProtocol and/or other telephonic device(s). Any device(s) which may beused to communicate using voice (speech) may be a user system, asearcher system and/or an ambassador system.

The search system 130 may include hardware and/or software interface toa system which provide communication services such as Instant Messaging(IM), email, Short Messaging Service (SMS), Enhanced Messaging Service(EMS), Multimedia Messaging Service (MMS), and/or other forms ofmessaging services. Any device which may communicate using such servicesmay be a user system, a searcher system, and/or an ambassador systemwithin the scope of the disclosure herein. A request may be submitted tothe search system 130 using any or all communication services which areable to communicate with the search system 130.

Although the search system 130 is illustrated as a single system, thesearch system 130 may include any number of hardware systems, which mayfunction as servers or other elements. The search system 130 may includeone or more servers, computers, etc. For example, servers such as thePowerEdge® 2900 by Dell, or the BladeCenterJS22 by IBM, or equivalentsystems might be used to implement elements of the search system 130.The search system may utilize an operating system (OS) such as MicrosoftWindows XP, or Linux, etc. Voice routing and packet switching may beaccomplished using well established technologies such as those providedby Cisco, or other networking companies. For example, a serverfunctionality associated with any of the communication servicesidentified herein above may be implemented to allow messages to betransmitted between the elements of the system 100. After beingpresented with the disclosure herein, one of ordinary skill in therelevant art will immediately realize that any viable computer systemsor communication devices known in the art may be used as user systems,guide systems, ambassador systems, and/or to implement the search system130.

The search system 130 is communicatively coupled with the database 120.As will be described herein in further detail below, the databaseincludes data that is processed during operation of the embodiments.Although FIG. 1 illustrates the database 120 as a separate component ofthe system, the database 120 may be integrated with the alternate searchsystem 130. Further, the records maintained in the database 120 may bestored in any typical manner, including in a Network Attached Storage(NAS), a Storage Area Network (SAN), etc. using any typical orproprietary database software such as DB2®, Informix®, Microsoft®SQLServer™ MySQL®, Oracle®, etc., and may also be a distributed databaseon more than one server. Elements of the database 120 may reside in anysuitable elements of the system 100. For example, elements of thedatabase 120 may be stored in any storage media accessible to a usersystem, a searcher system, and/or an ambassador system.

After being presented with the disclosure herein, one of ordinary skillin the art will readily appreciate that the computer systems of theembodiments can be any type of viable systems known in the art.

In the disclosed system and method, guides (searchers) and/orambassadors (expediters) may be required to register with the searchsystem 130, and in certain embodiments undergo training prior to beingpermitted to provide responses to requests submitted by a user utilizingthe search system 130. In one embodiment, a prospective guide may alsobe invited to register as a guide with the search system 130. In such asystem, a previously registered guide and/or an affiliate group mayinitiate the invitation process and thereby be designated as a sponsorfor the invited guide. In one embodiment of the disclosed system andmethod, a prospective guide is sent an email invitation to register as aguide with the search system 130. The email invitation in one embodimentcontains a link to a guide registration page and some indiciarecognizable by the search system 130 as to the person or entity thatinvited a prospective guide to register so that upon registration theperson or entity may be accredited with sponsorship of the invitedguide.

Should a prospective guide elect to register as a guide with the searchsystem 130, the prospective guide may for example initiate aregistration process by selecting a link that directs the guide's webbrowser to a URL of a guide registration page generated by a web serverfunction of the search system 130 (FIG. 1). While registration of aprospective guide is described with respect to the search system 130providing pages to a browser, the present invention is not limited toany particular configuration for registering guides. For example, asystem dedicated to registration of guides may be provided which isseparate from the search system 130.

A prospective guide may elect to be a searcher and/or an ambassador,and/or to perform other activities. A prospective guide may undergotraining and/or testing which may be used to determine whether a guideis competent to perform a task. After completion of testing and/ortraining, a guide may be allowed to perform a task. A registration andqualification process for a guide(s) is further described herein below.A guide may act as an ambassador, and/or as a searcher, etc. based oncompletion of training and/or testing. An ambassador processes a usersearch request and/or a search result(s). A searcher may receive aprocessed request and may perform a search responsive to the processedrequest. Various criteria may be considered for determining whether aguide may act as an ambassador and/or a searcher including but notlimited to time spent as a guide, level of knowledge in particular area,etc. In at least one embodiment, the role of ambassador and searcher maybe mutually exclusive.

The performance of an ambassador may be monitored. Multiple feedbackmechanisms may be utilized including ratings by one or more users,guides, or other persons, time required to complete a task(s),complexity rating of a query, and/or other speed and/or quality metrics.In at least one embodiment, a rating of an ambassador may be based on apercentage of queries provided to an ambassador which are answeredwithout utilizing a searcher.

As illustrated in FIG. 2, a process 200 for performing an informationsearch in which one or more ambassadors act as an intermediary isprovided. The process 200 may be operative on any suitable elements ofthe system 100. In at least one embodiment, the process 200 is operativeon a server associated with the search system 130.

In operation 205 (FIG. 2) a determination is made as to whether arequest is received. If it is determined in operation 205 that a requestis not received, control remains at operation 205 and process 200continues. If it is determined in operation 205 that a request isreceived, control is passed to operation 210 and process 200 continues.

The determination in operation 205 may be made using various criteria.In at least one embodiment, if a message is received at a serverassociated with the search system 130, it may be determined that arequest is received. For example, if an email message, an SMS, EMS,and/or MMS message, an IM, an IP message, and/or a voice message isreceived at an address associated with the search system 130 (FIG. 1) itmay be determined that a request is received. The search system 130 maydetermine whether the assistance of a guide will be used to respond to arequest as further described herein.

In operation 210 an ambassador processes a request. A process foraugmenting, modifying or processing information of a request using anambassador is further described herein with respect to FIG. 3. Controlis passed to operation 215 and process 200 continues.

In operation 215 a determination is made as to whether a processedrequest is accepted. If in operation 215 it is determined that aprocessed request is not accepted, control is passed to operation 210and process 200 continues. If in operation 215 it is determined that aprocessed request is accepted, control is passed to operation 220 andprocess 200 continues.

The determination in operation 215 may be based on various criteria. Inat least one embodiment, a request modified by an ambassador may beautomatically accepted if the ambassador indicates completion of a task.In at least one embodiment, a user may determine that a processedrequest is accepted. In at least one embodiment, a searcher maydetermine that a processed request is accepted. An ambassador may beassociated with a rating and/or compensation structure. The ratingand/or compensation of an ambassador may be affected by acceptance of arequest processed by the ambassador. If a vetted query is not acceptedby a searcher, it may be assigned to a second ambassador. Actions ofguides may be utilized as a form of quality control. For example, if twoambassadors agree on the content of a vetted query, and a secondsearcher accepts a vetted query, the rejected vetted query may not becounted towards the rating and/or compensation of an ambassador. Thecompensation and/or rating of a searcher who rejected a vetted query maybe affected if a rejected vetted query is accepted by an ambassadorand/or a different searcher.

In operation 220 an information search is performed by a searcherresponsive to a vetted query and a search result is returned. Aninformation search may be performed by any guide. A searcher and/or anambassador may perform a search. Control is passed to operation 225 andprocess 200 continues. Alternatively, in operation 220, an automatedsearch without requiring a searcher may be performed responsive to themodified request.

In operation 225 a search result is (optionally) reviewed by anambassador and may be presented to a user. A process for review andpresentation of a search result is further described herein below withrespect to FIG. 4. In at least one embodiment, a search result may besent to a user without requiring review by an ambassador. Control ispassed to operation 230 and process 200 continues.

In operation 230 rating information is obtained. A rating may be basedon user acceptance, and/or a review by a guide, and/or review by asystem administrator. Information of the process 200 is recorded.Information such as ratings of a search result, ratings of a guide,ratings of a vetted query, time to perform operations, etc. may berecorded. For example, the database 120 (FIG. 1) may be updated toindicate actions of a user, a searcher, an expediter, processing of aquery, etc. Control is passed to operation 205 and process 200continues.

In at least one embodiment, an ambassador who performs the processing inoperation 210 and an ambassador who performs the review in operation 225may be the same person. In other embodiments, a different ambassador mayperform the processing and the review operations. In at least oneembodiment, a searcher may perform the review operation.

As illustrated in FIG. 3, a process 300 for providing a response to arequest is provided. The process 300 may be operative on any suitableelements of the system 100. In at least one embodiment, the process 300is operative on a server associated with the search system 130.

In operation 305 a determination is made as to whether a guided requestis received. If it is determined in operation 305 that a guided requestis not received, control remains at operation 305 and process 300continues. If it is determined in operation 305 that a guided request isreceived, control is passed to operation 310 and process 300 continues.

The determination in operation 305 may be made based on variouscriteria. For example, a user preference may indicate that a guidedsearch is to be performed. The search system 130 (FIG. 1) may determinewhether a guided request is received. Content of a message may be usedto determine whether a guide is to respond to a request. A targetaddress such as a telephone number, a short code, an IM credential, etc.may be used to determine whether a guide is to respond to request. Anautomated analysis of a query may be performed as further describedherein which may in part determine whether a guide will respond to arequest.

In operation 310, a request (raw query, or original) is processed todetermine relevant information. For example, a raw query may beprocessed to improve matching of the raw query to a database of queries.A raw query may include text, audio, and/or other media. A raw query maybe processed in order to determine one or more keywords, categoriesand/or vetted queries which may be associated with a raw query. A vettedquery is further described herein below. A user ID may be associatedwith a raw query, and a unique ID may be assigned to a raw query. A rawquery may be compared to structured queries using stemming, word counts,and/or other forms of processing which are further described herein.Control is passed to operation 315 and process 300 continues.

In operation 315 a determination is made as to whether an ambassador isavailable to respond to a guided search request. If in operation 315 itis determined that an ambassador is not available to respond to a guidedsearch request, control is passed to operation 315 and process 300continues. If in operation 315 it is determined that an ambassador isavailable to respond to a guided search request, control is passed tooperation 320 and process 300 continues.

The determination in operation 315 may be based on various criteria. Inat least one embodiment, any available ambassadors are notified, and afirst ambassador to respond is selected. If an ambassador is notifiedand does not respond, the ambassador may be recorded as unavailable. Inat least one embodiment, an ambassador may be selected based oninformation associated with a raw query. For example, an ambassador maybe selected based on a type of device a requester is utilizing to submita request, and/or to receive a search result. An ambassador may beselected based on a competency, a training status, a keyword and/orcategory, geographic information, demographic information, personalityinformation, interest area information, affiliate group information, acontractual relationship, a language skill, etc., which may beassociated with a raw query and/or an ambassador. If an ambassador iscurrently responding to a request, the ambassador may not be availableto process another request. If an ambassador is not logged-in theambassador may not be available. Availability of an ambassador may bedetermined dynamically. For example, if no ambassador who meets a groupof criteria is available, one or more criteria may be modified so thatan ambassador who meets a different group of criteria is available.Alternately, a system may wait until an ambassador who meets a group ofcriteria is available. An ambassador may be notified of a request viavarious forms of communication such as IM, a voice message, a pop-up ina browser, an email message, an SMS, EMS or MMS, etc. which may be usedto communicate with a system associated with the ambassador.

In operation 320 information of a raw query and/or associatedinformation such as a keyword, category, profile, vetted query, etc.identified in operation 310 are provided to an ambassador. An ambassadormay be presented with a GUI such as the GUI 1200 (FIG. 12) which mayprovide information of a request. Control is passed to operation 325 andprocess 300 continues.

In operation 325 an ambassador constructs a structured query. Astructured query may be based on a keyword, category, image, audio,video, a profile, a structured query, a vetted query and/or anyinformation associated with a raw query. An ambassador may select anadvertisement which may be transmitted to a user. Control is passed tooperation 330 and process 300 continues.

A vetted query be constructed by various construction methods. In atleast one embodiment, a structured grammar is employed which includesphrasing a request in the form of a question. A structured queryconstruction may require standard elements to be selected. Using theexample illustrated in FIG. 5, the structured query ‘Who is the startingquarterback for the Chicago Bears’ is a structured query associated withthe raw query ‘Who is the #1 quarterback for the Bears?’. An ambassadormay communicate with a user using a real-time interface such as a chator IM session, voice connection, text-to-speech, etc. Interaction with auser may allow an ambassador to determine information of a raw queryincluding intended meaning of a keyword, a geographic location, and/orother information which may be used to form a structured query and/orwhich may be associated with and/or contained in a vetted query. Anambassador may confirm user acceptance of a structured query which isassociated with a raw query. An ambassador may modify, adjust, qualifyand/or supplement a keyword and/or other information of a raw query. Forexample, if the query ‘best apple’ is a raw query, an ambassador mayconstruct the structured query ‘what is the best apple computeravailable?’, or the structured query ‘what is the best apple for bakinga pie?’, or the structured query ‘what is the best apple?’ Although avetted query may be ambiguous, it is structured to conform to a standardformat, which may improve matching of raw queries to structured queries.

An ambassador may be presented with a query template which requiresselection of a number of elements of a structured query from a list ofavailable alternatives. In at least one embodiment, an interrogatoryword or phrase such as ‘what, who, when, where, how many’, etc., may beselected from a list, and/or a verb or verb form such as ‘is, was, willbe’, etc., may be selected from a list. A structured query may beanalyzed automatically to check for conformance to spelling andgrammatical conventions of a target language. A structured or vettedquery may allow a searcher to receive a query which contains a standardset of information from an ambassador. A vetted query may format arequest for submission to an automated search system. An ambassador mayselect elements of a structured query based on a keyword and/or categoryassociated with a raw query. A raw query submitted by a user such as‘bear's quarterback’, or ‘QB for the bears’ may for example be convertedto a succinct or structured query such as ‘Who is the startingquarterback for the Chicago Bears?’. One or more previously constructedstructured queries associated with a raw query may be presented to anambassador as further described herein.

In operation 330 (FIG. 3) a request is associated with a number ofcategories. A categorization may be automatically associated with theraw query. An ambassador may be presented with a list of categorizationswhich may be associated with a request. In at least one embodiment, anambassador is not allowed to change a categorization associated with astructured query. For example, a categorization of a query reviewed by anumber of ambassadors and/or searchers may not be permitted to bechanged by a particular ambassador. Alternately, an ambassador may beallowed to change a categorization associated with a structured querybased on various criteria. An ambassador may be allowed to change acategorization of a structured or vetted query based on a ranking,rating and/or other characteristic associated with an ambassador and/ora structured query. For example, an ambassador who has correctlycategorized a number or a percentage of succinct or structured queriesmay be allowed to modify a categorization associated with a structuredquery. If a structured query has not been categorized more than a numberof times by an ambassador, an ambassador may be allowed to modify acategorization associated with the structured query. For example, if astructured query is associated with a category three times a change ofcategorization may not be allowed subsequently. A structured query whichis brief such as ‘Who are the Bears?’ may be allowed to be associatedwith multiple categories. A structured or “succinct” query which is morespecific such as ‘Who is the current quarterback for the University ofCalifornia Bears?’ may be allowed to be associated with only onecategory. Selection or creation of a structured query may change theranking of a category associated with a request.

An ambassador may be presented with a taxonomy tree structure whichallows an ambassador to modify a categorization of a request. In atleast one embodiment, a categorization selected by an ambassador may beused to determine a most likely categorization of a vetted query.Control is passed to operation 335 and process 300 continues.

In operation 335 an ambassador selects information (a search resultand/or other response) which may be presented to a user. For example, anambassador may select an advertisement to be presented to a user. Aselection may be based on an explicit selection by a guide, and/or maybe based on a number of categories selected by a guide. A categoryassociation may be used to select a guide and/or information to bepresented to a user and/or a guide such as an advertisement, a searchresource, and/or a search result. In at least one embodiment, anambassador may be presented with one or more search results which havebeen associated with a vetted query based on a ranking of a searchresult. A search result associated with a vetted query may be selectedin various ways. A vetted query may be associated with a human-touched(reviewed) search result which has been judged superior to one or morealgorithmic search results based on voting by one or more guides. Avetted query may be associated with a search result produced by analgorithmic search engine. A vetted query may be associated with asearch result produced by an affiliate group. An ambassador may select asearch resource which may be provided to a guide and/or a user. A searchresult may be automatically provided to a user based on the associationof a search result with a vetted query. A search result may be selectedby an ambassador. An ambassador may select to transmit any or allinformation associated with a vetted query to a user. An exemplary GUI1400 for selection of a response is illustrated in FIG. 14. Control ispassed to operation 340 and process 300 continues.

In operation 340 a determination is made as to whether a response isaccepted. If in operation 340 it is determined that a response isaccepted, control is passed to operation 355 and process 300 continues.If in operation 340 it is determined that a response is not acceptedcontrol is passed to operation 345 and process 300 continues. In atleast one embodiment, acceptance of a response or result may bedetermined based on user actions such as activating an action button,clicking on a link associated with a result, and/or sending a textmessage. In at least one embodiment, if an ambassador selects a searchresult, it is determined that a response is accepted.

In operation 355 a rating of an ambassador is determined. A rating of anambassador may be determined based on factors such as user acceptance ofa search result provided, time required to send a search result to auser, a number of structured queries submitted to a user for review,click-through of an advertisement, a rating by a user a guide, a systemadministrator, etc. For example, if a user accepts a first vetted queryproposed by an ambassador, an ambassador may receive a high rating, orif a user ‘clicks through’ a search result or advertisement provided byan ambassador, the ambassador may receive a high rating. In at least oneembodiment, an ambassador rating may be based on acceptance of a vettedquery by a searcher. In at least one embodiment, an ambassador ratingmay be based on the time between receiving a query and when a searchresult is accepted. In at least one embodiment, an ambassador rating maybe based at least in part on a percentage of search queries whichrequire a search by a guide other than the ambassador. A rating of anambassador may be based on a combination of ratings from a number ofsearch sessions, and/or rating factors. An ambassador rating may bebased on any information indicated in the database 120 (FIG. 1).Information of the process 300 is recorded. Information of a structuredquery, a raw query, a vetted query, a category, a keyword, anambassador, a searcher, a search result, an advertisement, and/or a usermay be recorded and/or modified. Control is passed to operation 305 andprocess 300 continues.

In operation 345 a searcher is selected. An ambassador may selectattributes of a searcher to provide a search result responsive to aquery or request. An ambassador may be presented with informationregarding searchers available to respond to a request. An ambassador maycommunicate with a user to modify criteria used to select a searcher.For example, an ambassador may determine time, rating, ranking,affiliation, demographic, geographic and/or other criteria for selectionof a searcher. In at least one embodiment, an ambassador may determinewhether an immediate response is required by a user in order to select asearcher. In at least one embodiment, an ambassador may select asearcher based on information such as a rating of search result whichhas been provided by a searcher. A searcher may be selectedautomatically. For example, a searcher may be selected by the searchsystem 130 (FIG. 1). Control is passed to operation 350 and process 300continues.

In operation 350 a search is performed by a searcher. A searcher may bea selected searcher. A rating(s) of an ambassador may be determined. Arating(s) of an ambassador may be based on various criteria as describedherein above. For example, actions of a selected searcher may influencea rating of an ambassador that responded to a request. In at least oneembodiment, a search result is provided by a searcher. Control is passedto operation 355 and process 300 continues.

As illustrated in FIG. 4, a process 400 for reviewing and/or modifying asearch result is provided. The process 400 may be operative on anysuitable elements of the system 100. In at least one embodiment, theprocess 400 is operative on a server associated with the search system130. In at least one embodiment, a searcher may perform a review of asearch result.

In operation 405 a determination is made as to whether a search resultis received. If in operation 405 it is determined that a search resultis not received control remains at operation 405 and process 400continues. If in operation 405 it is determined that a search result isreceived control is passed to operation 410 and process 400 continues.

In operation 410 information of a search result is presented for reviewby an ambassador who may also be a guide and/or searcher. In oneembodiment, the guide is a first available ambassador selected by thesearch system 130 (FIG. 1). In other embodiments, a guide may beselected based on criteria such as a rating, a skill set of a guide, acapability(ies) of a searcher system such as the ambassador system 145(FIG. 1), or other information indicated in the database 120.Information of a search result may be presented to a guide using a GUIsuch as the GUI 1300 illustrated in FIG. 13. Control is passed tooperation 415 and process 400 continues.

In operation 415 a guide reviews information of a search result. Areview may be based on criteria such as relevance, credibility,completeness, reference type, media type, etc. Control is passed tooperation 420 and process 400 continues.

In operation 420 a search result is modified and transmitted to a user.Modification of a search result may be performed according to variouscriteria. In at least one embodiment, a search result is restructured toa standard search result format. An initial structured search result maybe presented to an ambassador, based on an automated analysis of asearch result. A search result may be modified for various reasons. Forexample, an ambassador may elect to send only one search result. Thismay for example be due to a constraint of a user system. For example, ifa user is utilizing a messaging system such as SMS or other messagingservices such as multimedia messaging service (MMS), a search result maybe modified to be suitable for the capabilities of a user device.Likewise, if a user is utilizing a voice-based device, an ambassador maymodify a search result to be suitable for voice communication such asVoIP, text-to-speech, or other voice based systems. An ambassador maysend a search result to more than one user device. For example, an SMSresult may be sent to a text-based user device, and a search result maybe posted to a web page of the search system 130 (FIG. 1) which may beaccessed by a user at a later time, or a voice reply may be transmitted,and a text reply may be sent. An ambassador may determine a format forreplying to a user based on interaction with a user. An ambassador mayselect an advertisement to be sent to a user. An exemplary GUI 1500 forreviewing, modifying and transmitting a response to a user isillustrated in FIG. 15 Control is passed to operation 425 and process400 continues.

In operation 425 a rating may be obtained. A user rating of a searchresult, an ambassador, and/or an overall rating may be obtained. Arating of an ambassador may be determined based on various factors suchas time from receipt of a search result by an ambassador until a searchresult is sent to a user, number of search results sent to a user,number of search results accepted by a user, selection of anadvertisement sent by an ambassador to a user, selection of a searchresult sent to a user by an ambassador, subsequent usage of the searchsystem 130 by a user associated with an ambassador, number of searchresults requiring the use of a searcher other than the ambassador, etc.Control is passed to operation 430 and process 400 continues.

In operation 430, information of the process 400 is recorded. In atleast one embodiment, process information is recorded in the database120 (FIG. 1). Information associated with ratings provided by a user, asearcher and/or ambassador rating and/or ranking, delivery of a searchresult, actions of a user, a searcher, an ambassador, etc., may berecorded. Control is passed to operation 405 and process 400 continues.

When a request is received, content of the request may be recorded.Information associated with processing of a request may be recorded forvarious purposes. While specific examples of information which may berecorded are used for the purposes of illustration, other informationmay be recorded, and equivalent ways of storing and accessing which arewell known in the art may be used to implement the systems and methodsdescribed herein.

As illustrated in FIG. 5, a sample of a raw query record, of which oneor more may be associated with or resident in the database 120 (FIG. 1)is provided. The raw query record 500 may include a query ID field 505,a keyword field 510, a category field 515, a structured query field 520,a user list field 525, a profile field 530, a search resource ID field535, a search result ID field 540, and a guide ID field 545.

A raw query record may be created by various processes including aguided search session, an interactive training session, anon-interactive training session. A raw query record may be importedfrom an external resource accessible to the search system, etc. In atleast one embodiment, a raw query record is created if a request isreceived by the search system 130 (FIG. 1). A raw query (or originalquery) is a query which has not been processed. A raw query may includea fully-formed question/sentence, a keyword and/or a search phrase. Araw query may include media, including images, audio, video, etc., andmay include information such as a phone number, an IM credential, and/orother data which is associated with a user system and/or communicationservice.

The raw query ID field 505 preferably contains a unique identifier of araw query, which is preferably used consistently. For example, in atleast one embodiment, the raw query ID field 505 can include a randomlygenerated numerical code, and/or a text string indicating the content ofa raw query. A query ID serves to distinguish a raw query recordassociated with a raw query from a raw query record associated withother raw queries. Although particular examples of identifiers aredescribed herein, other types of identifiers uniquely indicating a querymay be utilized without departing from the spirit and scope of theembodiments herein.

A unique identifier is assigned to a raw query when it is entered intothe database 120 (FIG. 1). If it is determined that a raw query isidentical to an existing raw query in the search database, that querymay be assigned the same query ID and may be added to the raw queryrecord associated with the query ID. A request may include otherinformation than that explicitly provided by a user. For example,location information, profile information, etc., may be associated witha query which might differentiate the query from a raw query in thedatabase 120 (FIG. 1). A query may receive a new query ID and create anassociated raw query record based on information indicated by a guide.In at least one embodiment, each request or raw query is assigned aunique ID. As illustrated in FIG. 5, the string ‘Who is the #1quarterback for the Bears?’ is the query ID for the raw query record500.

The raw query keyword field 510 may include information of a keywordassociated with a raw query. Content of the raw query keyword field 510may be created by an automated processing or parsing of a raw query. Ahuman guide may select a keyword which is associated with a raw query.Processes for making and/or reviewing an association of a keyword with araw query are further described herein. Using the example illustrated inFIG. 5 the keyword ‘bears’ and the keyword ‘quarterback’ are associatedwith the request ‘Who is the #1 quarterback for the Bears?’ Keywordsassociated with a request may be ranked, and may be presented to anambassador in an order based on the ranking. Keywords associated with arequest may be used to rank information associated with the request.

The raw query category field 515 may include information of a categoryassociated with a raw query. Content of the raw query category field 515may be created by an automated classification of a raw query. A humanguide may select a category which is associated with a raw query.Processes for making and/or reviewing an association of a category witha raw query is further described herein. For example, in the recordillustrated in FIG. 5, the categories ‘Sports>News>NFL’ and‘Sports>NCAA>Football’ are associated with the raw query ‘Who is the #1quarterback for the Bears?’ A category associated with a raw query maybe used to rank information which may be selected responsive to thequery. For example, if the category ‘Sports>News>NFL’ is associated witha query, guides(s) may be selected to respond to the query based onrankings associated with the category ‘Sports>News>NFL’.

The raw query structured query field 520 may include information of astructured query associated with a raw query. The raw query structuredquery field 520 may further include rating information of a structuredquery with respect to a raw query. A ranking or rating of a structuredquery may affect an order in which a structured query is presented to aguide when constructing a structured query. Using the example in FIG. 5,three structured queries have been associated with the raw query ‘Who isthe #1 quarterback for the Bears?’ The structured query, ‘Who is thequarterback for the Bears?’, the structured query, ‘Who is the startingquarterback for the Chicago Bears?’ and the structured query, ‘Who isthe current quarterback for the University of California Bears?’ areassociated with the raw query ‘Who is the #1 quarterback for the Bears?’A structured query may be rated in various ways. For example, a morespecific structured query such as ‘Who is the current quarterback forthe University of California Bears?’ may be rated higher than a lessspecific structured query such as ‘Who is the quarterback for theBears?’ Alternately, a structured query may be rated based on frequencyof use in a given time interval. For example, if the structured query‘Who is the starting quarterback for the Chicago Bears?’ has beenassociated with the raw query ‘Who is the #1 quarterback for the Bears?’by ambassadors sixty percent of the time in the last twenty-four hours,it might be ranked higher than another structured query which has beenassociated with the raw query ‘Who is the #1 quarterback for the Bears?’by vetting guides twenty percent of the time in the last twenty-fourhours. A structured query might be rated based on keywords associatedwith a raw query. For example, if a user has submitted multiple querieswhich have been associated with a keyword, a structured query associatedwith that keyword might be rated higher. A structured query may be ratedbased on a category associated with a raw query. A structured query maybe rated based on a combination of ratings. A method for ranking ofstructured queries associated with a request is further describedherein.

The raw query user list field 525 may include information of a user thatsubmitted a raw query. The user list field 525 may be used to associatea user with a raw query. A raw query record may be created or modifiedwhen a user is associated with a raw query. Using the exampleillustrated in FIG. 5 the user ‘Bill1023’ is associated with the rawquery ‘Who is the #1 quarterback for the Bears?’. Using the associationof a user with a raw query, other information associated with a user maybe presented to a guide.

The raw query profile field 530 may include information of a profileassociated with a raw query. The raw query profile field 530 may be usedto associate a profile, which may contain additional information such asgeographic data, demographic data, personality data, time data, locationbased data, user system data, etc., with a raw query. A raw query recordmay be created or modified when a profile is associated with a rawquery. Using the example illustrated in FIG. 5 the profile‘Bill1023_profile’ has been associated with the raw query ‘Who is the #1quarterback for the Bears?’ This may indicate that profile informationassociated with the user ‘Bill1023’ is associated with the raw query‘who is the #1 quarterback for the Bears?’ A profile associated with araw query may be used to modify processing of a query, to rank an itemwhich is to be associated with a raw query or request, to targetadvertisements, etc.

The raw query search resource ID field 535 may include information of asearch resource associated with a raw query. Information of a searchresource associated with a raw query may be used to locate searchresources which may be provided to a guide in order to obtain a searchresult responsive to a request. In at least one embodiment, a searchresource is associated with a category. Using the example in FIG. 5, theresources ‘Sporting News’, ‘NFL Network’ and ‘Chicago SunTimes’ may beassociated with ‘Sports>News>NFL’, and NCAA News' is associated with‘Sports>NCAA>Football’. A resource may have a rating associated with acategory, which may be used to rank a resource. A ranking of a resourceassociated with a category may affect a probability that a resource willbe presented responsive to a request or raw query associated with thecategory.

The raw query search result ID field 540 may include information of asearch result associated with a raw query. Information of a searchresult associated with a raw query may be used to provide a searchresult to a user. For example, the result ‘Kyle Orton is the QB-GoBears’ might indicate a text message to be provided to a user via SMS,and the result <www.nfl.com/kyleorton> might be provided using a webserving functionality to a browser of a user device. Any number ofsearch results may be associated with a request. A search result may beassociated with a request automatically and/or using the assistance of aguide. In at least one embodiment, a ranking of a search resultassociated with a structured query may affect the probability that asearch result will be associated with a request associated with thestructured query.

The raw query guide ID field 545 may include information of a guideassociated with a raw query. Information of a searcher associated with araw query may be used to determine compensation for a guide. Forexample, a number of queries processed by an ambassador and/or asearcher may be used to determine compensation for the guide. Similarly,if a search result provided by a guide is reused, the guide may becompensated. Using the example in FIG. 5, the guides ‘Ambassador1000’and ‘Searcher22’ are associated with ‘Who is the #1 quarterback for theBears?’

A vetted query may be used to improve matching of a raw query. Forexample, a raw query may include errors, slang, anaphora, and/or otherforms of natural language information which may be difficult tointerpret using automated processing, but which may be readilyunderstood by a person. An ambassador and/or a guide may select a vettedquery which is associated with an answer in order to provide a responseto a request. A vetted query may unambiguously map to a response, and/ormay be used to assist a searcher to obtain a response.

As illustrated in FIG. 6, a sample of a vetted query record 600, ofwhich one or more may be associated with or resident in the database 120(FIG. 1) is provided. The vetted query record 600 may includeinformation of vetted query record including a vetted query ID field605, a keyword field 610, a category field 615, a raw query field 620, auser list field 625, a profile field 630, a search result ID field 635,and a searcher ID field 640

A vetted query record may be created by various processes, including aguided search session, an interactive training session, anon-interactive training session. A vetted query record may be importedfrom an external resource accessible to the search system 130 (FIG. 1),etc. In at least one embodiment, a vetted query record is created by anambassador. A vetted query may be associated with a number of searchresults. If a vetted query is selected, a search result and/or otherinformation associated with the vetted query may be provided responsiveto the selection. For example, if a vetted query is selected by anambassador responsive to a request, a search result associated with thevetted query may be provided to a user and/or the ambassador.

The vetted query ID field 605 preferably contains a unique identifier ofthe vetted query, which is preferably used consistently. For example, inat least one embodiment, the vetted query ID field 605 can include arandomly generated numerical code, and/or a text string indicating thecontent of the vetted query. A vetted query ID serves to distinguish avetted query record associated with a vetted query from a vetted queryrecord associated with other vetted queries. Although particularexamples of identifiers are described herein, other types of identifiersuniquely indicating a vetted query may be utilized without departingfrom the spirit and scope of the embodiments herein.

A unique identifier may be assigned to a vetted query when it is enteredinto the database 120 (FIG. 1). If it is determined that a vetted queryis identical to an existing vetted query in the search database, thatvetted query may be assigned the same vetted query ID and informationmay be added to the vetted query record associated with the vetted queryID. A vetted query may include other information than the explicitcontent of a request. For example, location information, profileinformation, etc. may be associated with a vetted query. For example, anidentical structured query such as ‘Where is a good place for dinner?’might be associated with multiple vetted queries depending on a profile(e.g. vegetarian, Muslim, Hindu) associated with the structured query.Information obtained by an ambassador may be used to differentiate onevetted query from another. As illustrated in FIG. 6, the string ‘Who isthe quarterback for the Chicago Bears?’ is the query ID for the vettedquery record 600 (FIG. 6).

The keyword field 610 for the vetted query may include information of akeyword associated with a vetted query. Content of the keyword field 610may be created by an automated processing of a query. Keywordsassociated with a vetted query may be used to rank the vetted query. Forexample, if a raw query includes a keyword associated with a vettedquery, the vetted query may be ranked higher and may be more likely tobe provided to a user, an ambassador, a searcher, etc. For example,synonyms of nouns and/or noun phrases indicated in a vetted query mightbe indicated in the keyword field 610. Using the example in FIG. 6, thenamed entity ‘Chicago Bears’ and the noun ‘quarterback’ are associatedwith the vetted query record 600.

The category field 615 of the vetted query may include information of acategory associated with a vetted query. If category associated with avetted query is associated with a raw query, the vetted query may beranked higher and may be more likely to be provided to a user, anambassador, a searcher, etc. For example, if a category is associatedwith a raw query automatically, a ranking of a vetted query associatedwith the category might be used to determine an order in which thevetted query is presented to an ambassador. Using the example in FIG. 6,the categories ‘Sports>News>NFL’ and ‘Sports>People>Chicago’ areassociated with the vetted query record 600.

The raw query field 620 of the vetted query may include information of araw query which have been associated with a vetted query. It may furtherinclude rating information of a raw query with respect to a vettedquery. The rating of a raw query associated with a vetted query maydetermine an order in which a vetted query may be presented to anambassador when constructing a structured query responsive to a userrequest for information. For example, if a raw query has been associatedwith a vetted query by ambassadors fifty percent of the time, the vettedquery may be presented higher in a list of suggested queries than avetted query which has been associated with a raw query by ambassadorstwenty percent of the time. A rating may be based on information such asratings associated with a guide making an association of a raw querywith a vetted query, voting by a guide associated with a category,keyword, profile and/or other information associated with a vettedquery, etc. In at least one embodiment, a rating of a raw queryassociated with a vetted query may indicate a probability that a rawquery is equivalent to the vetted query.

Using the example in FIG. 6, three raw queries have been associated withthe vetted query ‘Who is the quarterback for the Chicago Bears?’ The rawquery, ‘Who is the quarterback for the Bears?’, the raw query ‘Who isthe #1 quarterback for the Bears?’ and the raw query, ‘Chicago Bearsquarterback’ are associated with the vetted query ‘Who is thequarterback for the Chicago Bears?’ Such an association may be used toassist an ambassador in associating a raw query with a vetted queryand/or to perform other tasks such as selecting an advertisement, asearcher, etc.

The user list field 625 for the vetted query may include information ofa user that submitted a raw query which was associated with a vettedquery to the search system 130 (FIG. 1). The user list field 625 (FIG.6) may be used to associate a user with a vetted query. A vetted queryrecord may be created and/or modified when a user is associated with avetted query. Using the example illustrated in FIG. 6 the user‘Bill1023’ may have submitted the raw query ‘Who is the #1 quarterbackfor the Bears?’ which is associated with the vetted query ‘Who is thequarterback for the Chicago Bears?’. As illustrated in FIG. 6, it ispossible for more raw queries to be associated with a vetted query thanusers. For example, a phrase such as ‘Chicago Bears quarterback’ mightbe a phrase which was not submitted by a user, but which might be usedto rank a vetted query versus a raw query.

Association of a user with a previous query may be used to associate araw query submitted by a user with a vetted query. For example, a userwho previously submitted the raw query ‘Who is the #1 quarterback forthe Bears?’ which was associated with the vetted query ‘Who is thequarterback for the Chicago Bears?’ might submit a query such as ‘Whowas the punter for the Bears in 1960?’. As the previous raw query wasassociated with the vetted query ‘Who is the quarterback for the ChicagoBears?’ the new raw query might be associated with the vetted query ‘Whowas the punter for the Chicago Bears in 1960?’ Historical contextinformation may be used automatically and/or may be presented to anambassador and/or a searcher.

The profile field 630 of the vetted query may include information of aprofile associated with a vetted query by the search system 130 (FIG.1). The profile field 630 (FIG. 6) may be used to associate a profilewhich may include information such as geographic data, demographic data,personality data, time data, location based data, user system data, etc.with a vetted query. As previously mentioned, profile information maydifferentiate vetted queries which have similar query content. Forexample, if a raw query is associated with profile information, aranking of a vetted query may be affected by a match between the profileinformation associated with the vetted query and the profile informationassociated with the raw query. A search result which is providedautomatically when a vetted query is selected may be customized to auser profile. Profile information associated with a vetted query mightalso be used to determine information regarding a user. Using theexample in FIG. 6, the profile ‘Bears Fans’ is associated with thevetted query ‘Who is the quarterback for the Chicago Bears?’ Forexample, the association of the users ‘Bill1023’ and ‘Mark1000’ with thevetted query record 600 may be used to infer that the users are ‘BearsFans’ if multiple vetted queries associated with that profile areassociated with the users. Likewise, a comparison of the profile ‘BearsFans’ with profile information associated with ‘Bill1023’ and ‘Mark1000’may affect a ranking of the vetted query ‘Who is the quarterback for theChicago Bears?’ for a query submitted by ‘Bill1023’ and/or ‘Mark1000’

The search result ID field 635 for the vetted query may includeinformation of a search result associated with a vetted query. Searchresults may be rated in association with a vetted query. For example, anumber of guides and/or users may be presented with a query associatedwith the vetted query and one or more search results and may indicate anopinion regarding the search result. As illustrated in FIG. 6, thesearch results ‘Kyle Orton is the QB—Go Bears’, <www.nfl.com/kyleorton>,‘I am in love with Kyle Orton!’ and <www.kyleortonfans.com>areassociated with the vetted query ‘Who is the quarterback for the ChicagoBears?’ In at least one embodiment, a highest rated search resultassociated with a vetted query may be provided to a user if the vettedquery is selected. In at least one embodiment, search results associatedwith a vetted query may be presented to a guide in an order which may bebased on a ranking of the search results.

The searcher ID field 640 for the vetted query includes information of aguide associated with a vetted query. In at least one embodiment, asearcher is associated with a search result produced by the searcher.This may for example allow the search system 130 (FIG. 1) to track useof search result provided by a searcher. For example, a searcher mightprovide multiple responses to a vetted query in order that reuse of asearch result associated with the searcher may be increased. Likewise, asearcher might provide different results which might be delivered tousers using various communication services. For example, a searchermight provide a brief answer as an SMS message, a link to a URL for aweb browser, a mixed media message for email delivery, etc. Likewise, anidentifier of an ambassador who has utilized a vetted query might beindicated in the search result ID field 640. Using the exampleillustrated in FIG. 6, the searchers ‘Searcher22’ and ‘Searcher2242’ areassociated with the vetted query record 600. This may indicate that‘Searcher22’ and ‘Searcher'2242’ have provided a search resultassociated with the vetted query ‘Who is the quarterback for the ChicagoBears?’

In at least one embodiment, a raw query record such as the raw queryrecord 500 may be used to track information of a query as it isprocessed by the search system 130 (FIG. 1). For example, when a requestis received by the search system 130 a raw query record is created andthe query ID field 505 and the raw query user list field 525 and the rawquery profile field 530 are populated. A raw query may be processedautomatically, and content of the raw query keyword field 510, the rawquery category field 515, and the raw query structured query field 520determined and rated. Automated categorization and keyword weighting maybe performed. Matching to a database of structured queries may beperformed using a full-text search facility such as Lucene which mayperform stemming and fuzzy matching to determine a rating or probabilitythat a raw query matches a structured query as further described herein.

Subsequently, if an ambassador selects a structured query to associatewith a raw query, a search result may be associated with the raw query.Using the example in FIG. 6, if an ambassador elects to associate thevetted query ‘Who is the quarterback for the Chicago Bears?’ with theraw query ‘Who is the #1 quarterback for the Bears?’ that raw query isindicated in the raw query field 620 (FIG. 6). Search results associatedwith the vetted query record 600 may be associated with the raw queryrecord 500, as indicated in the raw query search result ID field 540. Anambassador may select one or more of the search results associated witha vetted query to be provided to a user. In such an instance, theresults selected may be indicated in the raw query search result IDfield 540. As the searcher ‘Searcher 22’ is associated with the searchresults ‘Kyle Orton is the QB—Go Bears’ and <www.nfl.com/kyleorton>, asindicated in the vetted query searcher ID field 640 (FIG. 6), the ID ofthe searcher may be added to the Guide ID field 545 (FIG. 5). Anidentifier of an ambassador may be added to the raw query record 500 inorder that ratings and/or compensation of an ambassador may bedetermined, as indicated in the raw query guide ID field 545. If nosearch result is associated with a vetted query, or if a rating of asearch result associated with a vetted query is below a predeterminedthreshold, a query may be assigned to a searcher, as described furtherherein. A searcher may be provided with search resources associated withone or more categories selected by an ambassador.

A vetted query may allow the search system 130 (FIG. 1) to provide anautomated response to a request. For example, if a vetted query isassociated with a search result, if an ambassador selects and/orconstructs a vetted query which is determined to match a vetted query inthe database a highest ranking search result associated with the vettedquery may be provided to a user.

A structured query may be used to obtain a dynamic search resultautomatically. For example, if a user asked ‘What did the Dow do today?’an ambassador might provide the structured query ‘What is the closingprice for the Dow Jones Industrial Average?’ which might produce aprogrammatic search result. Various forms of a natural languagequestions may be mapped by an ambassador to a vetted query, which mayimprove utilization of searchers and ambassadors.

By providing an ambassador guide, requests for which a response isreadily available may be handled quickly and efficiently. Use of anambassador and information included in a database of vetted queries mayincrease the number of automated responses, and/or may improve thetargeting of a request to a searcher, the content and clarity of arequest provided to a searcher and the selection of the search resourcesprovided to a searcher. Information of activity of ambassador guides maybe used to provide a database for processing of requests which mayimprove the performance of automated processing while providing humanintervention efficiently.

A process 700 for responding to a user request is illustrated in FIG. 7.The process 700 may be operative on any or all elements of the system100 (FIG. 1). In at least one embodiment, the process 700 is operativeon a server associated with the search system 130.

In operation 705 a determination is made as to whether a query (request)is received. If in operation 705 it is determined that a request is notreceived control remains at operation 705 and process 700 continues. Ifin operation 705 it is determined that a request is received control ispassed to operation 710 and process 700 continues.

In operation 710 Natural Language Processing (NLP) is applied to a rawquery. A raw query may have been converted from speech to text in orderthat NLP may be applied. An algorithm for NLP is further described withrespect to FIG. 8 herein below. Control is passed to operation 715 andprocess 700 continues.

In operation 715 a determination is made as to whether a request may beanswered automatically. If in operation 715 it is determined that arequest may be answered automatically control is passed to operation 770and process 700 continues. If in operation 715 it is determined that arequest may not be answered automatically control is passed to operation725 and process 700 continues. In at least one embodiment, it may bedetermined that a search result such as an advertisement, etc., may beprovided automatically while a search result which might include ananswer to a user query, a search resource, etc., might require furtherprocessing. In such an instance, control is passed to operation 770 andoperation 725 simultaneously.

In operation 725 an answer reuse function is calculated. A process andfunctionality for determining a probability that a reusable answerresponsive to a user request is available is further described hereinwith respect to FIG. 9. Control is passed to operation 730 and process700 continues.

In operation 730 a determination is made as to whether a reusable answeris available. If in operation 730 it is determined that a reusableanswer is available control is passed to operation 770 and process 700continues. If in operation 730 it is determined that a reusable answeris not available control is passed to operation 740 and process 700continues.

In operation 740 an expeditor usage function is calculated. A processand functionality for determining a probability that an expeditor is tobe used is further described herein with respect to FIG. 10. Control ispassed to operation 745 and process 700 continues.

In operation 745 a determination is made as to whether an expeditor isrequired to process a request. If in operation 745 it is determined thatan expeditor is required to process a request control is passed tooperation 750 and process 700 continues. If in operation 745 it isdetermined that an expeditor is not required to process a requestcontrol is passed to operation 765 and process 700 continues.

In operation 750 an expeditor processes a request. Processing of arequest by an expeditor is further described herein. An expeditor maycategorize a request, may form a structured query responsive to arequest, may identify a type of automated processing which may respondto a request, may select an answer to a user request, may designatelocation, profile, and/or other information associated with a request,etc. Control is passed to operation 755 and process 700 continues.

In operation 755 a determination is made as to whether an answer isavailable to a user request. If in operation 755 it is determined thatan answer is available to a user request control is passed to operation770 and process 700 continues. If in operation 755 it is determined thatan answer is not available to a user request control is passed tooperation 765 and process 700 continues.

In operation 765 a searcher performs a search for information andprovides a search result. A searcher may be selected based on anysuitable criteria. In at least one embodiment, a highest rankingsearcher associated with a category associated with a request isselected and is provided with search resources associated withinformation of the request such as a category, keyword, location,profile, etc. Control is passed to operation 770 and process 700continues.

In operation 770 a search result is provided to a user. A search resultmay be provided to a user using any communication service associatedwith a user. A search result which has been provided by an automatedresource, stored information, and/or by a human searcher is provided.Control is passed to operation 775 and process 700 continues.

In operation 775 information of the process 700 is recorded. Forexample, information of a user, a guide, a request, a resource, akeyword, a category, a vetted query, a profile, etc. which may have beenmodified by the process 700 may be recorded and/or updated. In at leastone embodiment, process information is recorded in the database 120(FIG. 1). Control is passed to operation 705 and process 700 continues.

As illustrated in FIG. 8 a process 800 for natural language based queryformation and improvement is provided. The process 800 may be referredto as the Re-Writing Process (RWP). The process 800 may be operative onany or all elements of the system 100 (FIG. 1). In at least oneembodiment, the process 800 is operative on a server associated with thesearch system 130 (FIG. 1).

In operation 810, a processing such as a signature stripping isperformed. Some queries such as SMS, MMS, email, or IM based queries mayinclude template information which must be removed in order toeffectively determine user intent. For example, a user may have asignature message which is added to an email or SMS message. As this iseasily detected by a person, if an expeditor identifies this content ina first message, subsequent messages may have the identified contentremoved without requiring the use of an expeditor. Likewise, standardemail signatures which contain elements such as phone numbers, etc.,might be detected and removed from a request message. In at least oneembodiment, if calling number identification (CNID) information isavailable, a comparison of the trailing characters of a number ofsuccessive messages may be used to determine if a signature exists. Forexample if 3 or more successive messages include a common characterstring at the end of the message, it may be determined that the commoncharacters represent a signature which is to be removed. Control ispassed to operation 815 and process 800 continues.

In operation 815 language detection is performed. Various types oflanguage detection algorithms may be applied. For example, a techniquedescribed in Language detection in unknown, International Conference onComputational Linguistics, Proceedings of the 18th conference onComputational linguistics—Volume 2 Saarbrücken, Germany SESSION: Projectnotes and demos Pages: 1021-1025, Year of Publication: 2000, relates toa typical algorithm, ora composite approach to language/encodingdetection, S Li, K Momoi—Proc. 19th International Unicode Conference,2001—sfr-fresh.com. A unique ID associated with a user, may be used torecognize an individual user, such an algorithm might be applied over aquery history, and/or may be influenced by factors such as profileinformation, area code, etc. which may be associated with a user.Control is passed to operation 820 and process 800 continues.

In operation 820 a spelling check is performed. A query string may be‘tokenized’ or separated into words based on white space indicators.Spelling check is performed using techniques such as word comparison andphrase comparison based on a language detected in operation 815. Iflanguage detection results were ambiguous, a spelling check versus morethan one dictionary may be performed, and language detection may berepeated based on results of probable spelling corrections. A spellingcheck may include variances based on a type of device associated with arequest. For example, a different correction algorithm might be appliedfor a query submitted using a 12-digit keypad, than a QWERTY keyboard.Control is passed to operation 825 and process 800 continues.

In operation 825 a grammar check is performed. Grammar check isperformed using well known techniques such as an annotation graph, whichare well known. For example ATLAS: A flexible and extensiblearchitecture for linguistic annotation, Steven Bird, David Day, JohnGarofolo, John Henderson, Christophe Laprun, Mark Liberman, submitted on13 Jul. 2000, describes a software tool for producing and usingannotation graphs. An annotation graph such as a Penn Tree may becreated wherein tokens are associated with parts of speech. Control ispassed to operation 830 and process 800 continues.

In operation 830 a word substitution is performed. Word substitution isthe process of replacing common abbreviations and/or slang forms ofspeech with an alternate form of a word or words. For example ‘b4’ mightbe replaced with ‘before’, or ‘u’ might be replaced with ‘you’, etc.Such a process may include factors which may be tailored to a user basedon factors such as profile information (e.g. regional information,demographic, etc.) and/or query history information, etc. Control ispassed to operation 835 and process 800 continues.

In operation 835 a query is formatted as a structured query. Such aprocess may be performed by semantic processing of a query. For example,a query may be created by converting a statement into a question. (e.g.‘will it rain today?’ might be constructed as ‘what is the chance ofrain today?’) Likewise, a raw query might be compared to a database ofraw queries which have been associated with a succinct query by aguide(s). An annotation graph may be used to format a raw query. Controlis passed to operation 840 and process 800 continues.

In operation 840 categories associated with a query are rated. Anexemplary algorithm for selecting and rating a categorization associatedwith a query is The Ferrety algorithm for the KDD Cup 2005 problem ZsoltT. Kardkov and Domonkos Tikk and Zolt, SIGKDD Explorer Newsletter,volume 7, number 2, 2005, issn 1931-0145, pages 111-116, ACM, New York,N.Y., USA and<http://www.cse.lehigh.eduhbrian/pubs/2007/query-classification/LU-CSE-07-013.pdf>by Bhandari and Davison. A taxonomy for categorization may be used whichis customized in order to index guides who may be associated withkeywords, topics, and/or categories. Training of a classificationalgorithm based on query classification information provided by guidesmay be used to improve classification performance. A keyword of a querymay be compared to keywords associated with an index of categories todetermine a category which is associated with a query. For example, if aquery includes a number of keywords which are associated with acategory, the category may be more likely to be associated with thequery. Likewise if queries associated with a category match a query, thecategory may be more likely to be associated with the query. Control ispassed to operation 845 and process 800 continues.

In operation 845 automated and/or keyword based queries may be excludedfrom processing. For example, if a keyword is associated with aparticular system response a query which begins with that keyword may beexcluded from processing. This might include advertising keywords,and/or other words and/or phrases which are associated with an automatedresponse. Likewise, a query which conforms to a query template after NLPmay be selected for an automated response. For example, if a usersubmitted the query ‘wether 46038’, the corrected version ‘weather46038’ would be automatically processed. Likewise a query such as ‘stoknfs’ which might be corrected to ‘stock nfs’ could be automaticallyprocessed. Control is passed to operation 850 and process 800 continues.

In operation 850 named entity processing is performed. Named entityprocessing techniques are well known in the art. For exampleIntroduction to the CoNLL-2002 Shared Task: Language-Independent NamedEntity Recognition, author Erik F. Tjong Kim Sang University of Antwerp,2002 describes a typical process and method for performing named entityprocessing. For example, the search system 130 (FIG. 1) may access adatabase of named entities which may be included in the database 120,and/or may be an external search resource. Control is passed tooperation 855 and process 800 continues.

In operation 855, information associated with the NLP operationsperformed in conjunction with the process 800 is recorded. For example,information of ratings of a category associated with a raw query, anannotation graph, named entity tags, etc associated with NLP processingof a raw query may be recorded in the database 120 (FIG. 1). Forexample, results of the processes, the confidence and/or probabilityfactors associated with various versions of a query, etc., which may beshared with other processes and/or may be used to evaluate training andefficacy of a particular parametric tuning, etc. may be recorded.Control is passed to operation 860 and process 800 continues.

In operation 860, information associated with a query is updated. Aquery record such as the raw query record 500 may be updated to reflectcategorizations, structured queries, keywords, and/or ratingsinformation determined based on the process 800. Control is passed backto the parent process and process 800 terminates.

As previously discussed, answer reuse is important to success of ahuman-assisted search system as it directly impacts cost and economicviability of the system. In order that a previous search result or anautomated search result may be provided responsive to a request, it isnecessary to meet a combination of various conditions. For example, arequest must be sufficiently similar to a request for which a searchresult is available. Likewise, an available search result must be ofsufficient quality to be provided to a user. Criteria for matching andcriteria for quality of a search result may change based on the natureof a request and a type of search result.

For example, a search resource, an advertisement, a game, an answer,contact information for a supplier of items or services, etc., may havedifferent quality and/or matching criteria. Quality rating of a resourcemight be based for example on reliability of providing information.Quality rating of an advertisement might be based on fit with a categoryand/or a payment by an advertiser. Quality rating of a game might bebased on a match to keyword and/or profile information associated withprevious purchases.

Similarly to a quality rating, matching to a query may have differentcriteria for different types of search results. For example a searchresult such as an advertisement might be provided if matching is lessexact, while a search result such as an answer to a question might beprovided if matching is more exact. To determine whether an answer is tobe reused, a function which describes a level of matching between aquery and a reference query with which a search result is associated isneeded. In at least one embodiment, software which implements anevaluation function is referred to as the “Answer Reuse Process” (ARP)and an ARP evaluation function is used to determine whether an answer isto be reused.

In at least one embodiment a process which provides a set of candidatequeries which match a reference query and answers associated with thecandidate queries is implemented. In at least one embodiment, the “QueryVetting Process” (QVP) compares any query to a database of queries andreturns any queries which match the reference query to a giventhreshold. For example, the QVP might return all queries which matchedany word in a query or any queries that match 80% of the words in aquery, etc. Techniques such as stemming, etc. may be utilized by the QVPand a matching algorithm may be adjusted to provide a manageable set ofqueries and associated answers to the ARP. An index of queries may beused to determine matching of queries to a reference query. For example,queries may be considered to be documents within a corpus, and a rankingof a query with respect to a reference query may be determined based onword frequency analysis. Word frequency analysis may for example includea number of times a word appears in a reference query and a number oftimes the word appears in a query to be ranked. Word frequency analysismay include a weighting based on frequency of a word within a corpus.For example, words such as ‘the’, ‘of’, etc. which may appear in a highpercentage of queries in a corpus of queries may have a low weighting,while words which appear with low frequency such as nouns, noun phrases,named entities, etc. may have a high weighting.

In at least one embodiment the ARP evaluator function which determinesthe probability that a search result will be reused is defined as

f _(arp)(a _(x) , q)=f _(qual)(a _(x))*f _(match)(q, q _(ax))   Eq. 1

Wherein:

f_(arp)(a_(x),q) is the ARP answer rating for a potential answer (a_(x))with respect to the query (q). The answer rating function f_(arp) isnormalized such that it as a value between zero and one inclusive. Inother words 0≦f_(arp)(a_(x), q)≦1

f_(qual)(a_(x)) is a quality rating associated with the answer a_(x)with respect to the query, q_(ax) that is associated with the answera_(x). The quality function f_(qual) is normalized such that it has avalue between zero and one inclusive. In other words 0≦f_(qual)(a_(x))≦1

f_(match)(q,q_(ans)) is the Query Match Function, defined below, whichrates the similarity between the reference query q and the queryassociated with the answer q_(ans). The match function f_(match) isnormalized such that it has a value between zero and one inclusive. Inother words 0≦f_(match)(q,q_(ans))≦1

An answer a_(max) is selected which has the highest value of f_(arp) forthe set of answers which are associated with queries which are providedto the ARP by the QVP. If the value of f_(arp) associated with theselected answer a_(max) is below a threshold, the answer will not bereused. Thus the probability that an answer will be reused is a functionof both the match between a query (e.g. a vetted query, a raw query,etc.) and the query associated with the answer and the quality ratingassociated with the answer and its related query.

The Answer Quality Function (AQF), f_(qual)(a), may be determined invarious ways. For example, an answer which has been selected by a highlyrated searcher may have a high value of f_(qual)(a). An answer which hasreceived a high percentage of positive ratings by users and/or guidesmay have a high value of f_(qual)(a). An answer which has been providedby an automated search may have a low value of f_(qual)(a). In at leastone embodiment, a discrete value function may be used to assignf_(qual)(a)=1 for a reviewed, superior, answer provided by a guide,f_(qual)(a)=0.7 for an answer which has received at least 80% positiveratings by users, and f_(qual)(a)=0.5 for an answer which has beenselected by at least one ambassador. The AQF may include other variablessuch as how often a result has been provided to users, time basedmodification, sponsorship factors, adjustment for characteristics of asearcher, etc.

A Query Matching Function (QMF) is used to determine how closely relateda test query is to a reference query. The QMF may be based on anyinformation associated with a query such as a category, a structuredquery, a raw query, a profile, a location, a language, etc. A QMF actsto normalize a ranking of queries returned by a database query from anindex such as that provided by the QVP. In at least one embodiment, theQMF is defined as:

$\begin{matrix}{{f_{match}( {q,q_{a_{x}}} )} = {{w_{wb}{f_{wb}( {q,q_{a_{x}}} )}} + {w_{pb}{f_{pb}( {q,q_{a_{x}}} )}} + {w_{a\; c}{f_{a\; c}( {C_{q},C_{q_{a_{x}}}} )}}}} & {{Eq}.\mspace{14mu} 2}\end{matrix}$

The reference query, q, is the query which is to be matched from Eq. 1.In at least one embodiment, q is a succinct query which has beenproduced by the RWP process 800 (FIG.

8). The category set C^(q) is the set of categories that have beenselected to be associated with the reference query q. In at least oneembodiment, C^(q) is the set of categories which have been determined bythe RWP process 800. The candidate answer a_(x) is an answer which isassociated with a set of vetted queries which has been selected forevaluation versus a reference query q. In at least one embodiment, thecandidate queries and associated answers are provided by the QVP. Thecandidate category set

C_(q_(a_(x)))

the set of categories associated with the query associated with thecandidate answer a_(x).

The QMF includes the weighting factors w_(wb), w_(pb), and w_(ac). Inorder to preserve normalization it will be readily appreciated that thevalues of f_(wb), f_(pb) and f_(ac) must be between zero and oneinclusive, and that w_(wb)+w_(pb)+w_(ac)=1. The components of the matchfunction f_(match) are the Wordbag Function (WBF), the PhrasebagFunction (PBF) and the Autocat Function (ACF). The WBF is used tocompare words and their grammatical functions between two queries. ThePBF is used to compare phrases and their grammatical functions betweentwo queries. The ACF is used to compare categories between two queries.

In at least one embodiment, the WBF compares the tagged word content ofthe reference query q and the query associated with the answer a_(x),q_(a) using the following formula:

$\begin{matrix}{{f_{wb}( {q,q_{a_{x}}} )} = \frac{{Q_{q}\bigcap Q_{{q_{a}}_{x}}}}{{Q_{q}\bigcup Q_{q_{a_{x}}}}}} & {{Eq}.\mspace{14mu} 3}\end{matrix}$

Q_(q) is the set of tagged tokens in the reference query q. A “taggedtoken” is a word along with a tag which identifies the part of speechwhich has been determined from semantic analysis of context of thequery. Tagged tokens are considered identical only if both the wordcontent and the part of speech tag are identical.

Q_(q_(a_(x)))

is the set of tagged tokens in the query associated with the answera_(x).

The Phrasebag Function (PBF) is similar to the Wordbag Function (WBF)but the PBF utilizes word phrases instead of words. W_(q) is the set oftagged phrases in the reference query q. A “tagged phrase” is a group ofwords along with a tag which identifies the part of speech associatedwith the group which has been determined from semantic analysis ofcontext of the query (e.g. noun phrase, prepositional phrase, etc.).Tagged phrases are considered to be identical if both the phrase contentand the associated tag are identical.

W_(q_(a_(x)))

is the set of tagged phrases in the query associated with the answera_(x). The PBF is defined as:

$\begin{matrix}{{f_{pb}( {q_{a_{x}},q} )} = \frac{{w_{q}\bigcap w_{q_{a_{x}}}}}{{w_{q}\bigcup w_{q_{a_{x}}}}}} & {{Eq}.\mspace{14mu} 4}\end{matrix}$

The WBF and the PBF may include consideration for synonyms and otherword relations as well as the frequency of various tagged phrases. Forexample, a group of queries provided by the QVP might be indexed basedon synonyms, and a WBF and PBF might be calculated based on the querieswith the synonyms substituted. In such an instance a query such as ‘whatis the most common bike?’ might cause the tokens ‘most common bicycle’,‘most common motorcycle’ and ‘most common cycle’ to be used to retrievecandidate queries from an index. A WBF and PBF calculation might weighta match to the ‘most common bike’ a higher weight than a match to ‘mostcommon bicycle’, etc. However, candidate queries which match the synonymtokens might be rated highly.

The Autocategorization Function (ACF) may be defined as:

$\begin{matrix}{{{f_{ac}( {C_{q_{a_{x}}},C_{q}} )} = {{if}\mspace{14mu} \begin{Bmatrix}{{C_{q}\bigcap C_{q_{a_{x}}}} = {\varnothing 0.0}} \\{{{C_{q}\bigcap C_{q_{a_{x}}}} \neq {\varnothing {f_{smooth}( {\max ( {{likelihood}(c)} )} )}}},} \\{\; {c \in \{ {C_{q}\bigcap C_{q_{a_{x}}}} \}}}\end{Bmatrix}}}\mspace{20mu} {{f_{smooth}(x)} = \frac{{\sin ( {x\; \pi} )} + 1}{2}}} & {{Eq}.\mspace{14mu} 5}\end{matrix}$

That is to say, if the intersection of the set of categories associatedwith candidate queries returned by the QVP with the set of categoriesreturned by the RWP is null, the value of the ACF is zero. If there arecategories in the intersection, the highest rating value of any categoryin the intersection is selected. The rating value associated with theselected category is modified by the smoothing function f_(smooth). Inembodiments, the WBF, PBF, and ACF may be ignored by setting theassociated multiplier to zero. Weighting of the WBF, PBF, and ACF may beadjusted based on whether a person such as an expediter, a user, asearcher, etc., has reviewed a vetted query, category, rewritten query,etc.

Actions of guides and users responsive to results produced by NLPprocessing may be used to tune various elements of the performance ofalgorithms. For example, data of acceptance of a structured query byexpeditors may be used to determine a weighting given to words and/orphrases when constructing a structured query. Similarly, languagedetection, named entity processing, spelling correction, etc., may bemodified based on actions of guides. Feedback activities by guides mayfor example allow NLP, speech recognition, and/or other forms ofautomated conversion of requests to be personalized to a user withoutthe need for a user to make such corrections.

As illustrated in FIG. 9 a process 900 for determining whether areusable search result as a response to a request is available isprovided. The process 900 may be operative on any or all elements of thesystem 100 (FIG. 1). In at least one embodiment, the process 900 isoperative on a server associated with the search system 130 (FIG. 1).

In operation 905 parameters for the reuse evaluation function(s) such asthose described herein above are determined. For example if the answerreuse function is to be evaluated (calculated) in operation 725 (FIG. 7)the parameters and/or the decision thresholds may be adjusted due to thelack of human intervention in the processing of the query. Likewise if aguide such as an expeditor, searcher, and/or transcriber has processed aquery, parameters and thresholds for query matching and answer reuseand/or automation may be modified. Control is passed to operation 910and process 900 continues.

In operation 910 a list (number) of structured or succinct queries whichmay be compared to a reference query and associated categories which areassociated with a raw query are obtained. In at least one embodiment,the QVP may provide such information to the ARP. In at least oneembodiment, candidate succinct queries may be selected based at least inpart on content of search results or answers associated with thesuccinct queries. For example, an index such as a Lucene® indexassociated with a corpus of succinct queries may be queried and a numberof the highest ranked queries associated with a raw query may beobtained. Control is passed to operation 915 and process 900 continues.

In operation 915 a ranked list (number) of structured or succinctqueries and categories which are associated with a query produced by NLPoperations such as those performed in the process 800 are obtained. Inat least one embodiment, the QVP may provide such information to theARP. Succinct queries may be selected based at least in part on contentof search results associated with the succinct queries. Control ispassed to operation 920 and process 900 continues.

In operation 920 the ARP evaluation function is evaluated for the set ofanswers associated with the structured queries and categories obtainedin operation 910 and 915. In at least one embodiment, the queries fromoperation 910 are weighted differently than the queries from operation915. In particular, if a structured query and a category have beendetermined by a guide and/or a user, answers and/or candidate queriesassociated with a raw query may be more or less likely to be selected.Information of the ARP evaluator function associated with the candidateanswer set is recorded. For example, the ARP evaluator function valuefor answers associated with the three highest ranked or rated vettedqueries may be recorded. Control is passed back to the parent processand process 900 terminates.

As illustrated in FIG. 10 a process 1000 for determining whether arequest is to be directed to an expeditor or ambassador is provided. Theprocess 1000 may be operative on any or all elements of the system 100(FIG. 1). In at least one embodiment, the process 1000 is operative on aserver associated with the search system 130 (FIG. 1).

In operation 1010 a function which measures a probability that acategory selected to be associated with a request is valid is evaluated.A function such as the ACF described herein above may be utilized. Acategory confidence function may include factors such as user history,user profile, guides associated with a user, etc. In at least oneembodiment the ACF is applied to a raw query and a succinct querycreated using the process 800 (FIG. 8). Control is passed to operation1015 and process 1000 continues.

In operation 1015 a determination is made as to whether an expeditor isrequired to select a category to be associated with a request. If inoperation 1015 it is determined that an expeditor is required to selecta category to be associated with a request control is passed tooperation 1020 and process 1000 continues. If in operation 1015 it isdetermined that an expeditor is not required to select a category to beassociated with a request control is passed to operation 1030 andprocess 1000 continues. The determination in operation 1015 may be basedon various criteria. For example, if the ACF value is below apredetermined value, it may be determined that an expediter is requiredto select a category associated with a request, or if an expeditor haspreviously categorized a query it may be determined that an expediter isnot required to select a category.

In operation 1030 a function (the Auto Guide Probability (AGP) function)which measures a probability that a programmatic (automatic) responsefrom a search resource may be used to respond to a request is evaluated.For example, a query may be analyzed to determine if it may be mapped toan API available to provide information such as weather, sports scores,etc. Likewise, keywords of a request may be examined in order todetermine the probability that a response from an expeditor can be used.For example a function such as the ARF evaluation function may beapplied to a query structure or template associated with a resource anda query. Items such as named entities detected in a query may be used todetermine a probability that an expediter may respond to a request usingan automated resource or ‘finder’ which utilizes stored informationand/or responds to information provided using an API. The AGP functionmay be applied to the process step 715 (FIG. 7). Control is passed tooperation 1035 and process 1000 continues.

In operation 1035 a determination is made as to whether an expeditor mayprovide a response to a request using a resource. If in operation 1035it is determined that an expeditor may provide a response to a requestusing a resource control is passed to operation 1020 and process 1000continues. If in operation 1035 it is determined that an expeditor maynot provide a response to a request using a resource control is passedto operation 1040 and process 1000 continues. For example, a thresholdvalue associated with a probability function such as the AGP may becompared to a computed value of the probability function for a requestto determine whether an expediter may provide a response to a requestusing a resource. In at least one embodiment, a threshold used inoperation 1035 may be lower than a threshold used in operation 715,which may be used to vary quality and/or cost of answering a query.

In operation 1040 a function which measures a matching range between asuccinct query and a request is evaluated. For example, a function suchas the WBF, PBF and/or ACF may be evaluated to determine if a candidatequery, such as a query indicated in the database 120 (FIG. 1) matches auser request, which may include a succinct query. For example, asuccinct query may exist which is not associated with an answer, and/orwhich is associated with an answer of low quality. Control is passed tooperation 1045 and process 1000 continues.

In operation 1045 a determination is made as to whether a matchingsuccinct query is known (available). If in operation 1045 it isdetermined that a matching succinct query is not available control ispassed to operation 1020 and process 1000 continues. If in operation1045 it is determined that a matching succinct query is availablecontrol is passed to operation 1050 and process 1000 continues. Athreshold applied to the value of the function computed in operation1040 may be used to determine whether a matching succinct query isavailable.

In operation 1050 a function which measures (calculates) the sensitivityof a request to location is evaluated. For example, if a user requestincludes a word which implies location (i.e. exophora) such as ‘here’,‘there’, etc. and/or includes a named entity such as a street name, acity name, etc., a location sensitivity function may be affected by suchcontent. For example, a query matching a user query may not beassociated with an answer due to ambiguity which may be resolved basedon location information. Likewise, a location sensitivity function maybe affected by the presence and/or absence of location informationassociated with a user associated with a request. For example, if aquery is associated with a reliable source of location information, anda query is for generic information, a value of a location sensitivityfunction may be reduced. Control is passed to operation 1055 and process1000 continues.

In operation 1055 a determination is made as to whether a query islocation sensitive. If in operation 1055 it is determined that a queryis location sensitive control is passed to operation 1020 and process1000 continues. If in operation 1045 it is determined that a query isnot location sensitive control is passed to operation 1060 and process1000 continues.

In operation 1060, information of a request is provided to a searcher. Asearcher may be selected based on the categorization, succinct query,and/or other information including data associated with the request. Itshould be noted that the cost of processing a query may be reduced as asearcher may be responding to well formed queries in the searcher's areaof interest, rather than responding to queries which may requirerevision, correction, etc. Control is passed back to the parent processand process 1000 terminates.

In operation 1020 information of a request is provided to an expeditor.A query has been evaluated to determine the nature of the request whichmay improve the probability that an expeditor may be selected moreeffectively. An expeditor may further be provided with a toolset whichis adapted to the most likely response type for a query. As such anexpeditor may be able to process a request more rapidly and with fewererrors. For example, if an expediter receives a request as a result ofthe determination in operation 1015, a categorization tool might beprovided first, while if an expediter is provided with a query as aresult of the determination in operation 1035, a tool for providing ananswer using the most probable automated resource may be provided.Control is passed back to the parent process and process 1000terminates.

As illustrated in FIG. 11 a process 1100 for qualifying a query by anambassador is provided. The process 1100 may be performed automatically,or may be performed using the assistance of a guide in any or alloperations. The process 1100 may be operative on any or all elements ofthe system 100 (FIG. 1). In at least one embodiment, the process 1100 isoperative on a server associated with the search system 130 (FIG. 1)and/or the ambassador system 145.

In operation 1110 a determination is made as to whether a request isreceived. If in operation 1110 it is determined that a request is notreceived control remains at operation 1110 and process 1100 continues.If in operation 1110 it is determined that a request is received controlis passed to operation 1115 and process 1100 continues.

In operation 1115 information of a query including a category and astructured query is obtained. Any information associated with a query,such as a user history, a raw query, a processing history, etc., may beobtained. In at least one embodiment, information of a request isprovided to an ambassador. Control is passed to operation 1120 andprocess 1100 continues.

In operation 1120, information of a query is analyzed for compatibilitywith available resource(s) which may provide a response. A function maybe evaluated to determine if a query may be answered by an automatedprocess. For example, if a query includes words, named entities,keywords, etc., such information may be evaluated. In at least oneembodiment, information of a query is provided to an expeditor who mayevaluate the ability to respond using a resource. An expediter maymodify a query, which may be reevaluated automatically and/or whentriggered by an expeditor. For example, the AGP function associated witha modified query may be evaluated. Control is passed to operation 1125and process 1100 continues.

In operation 1125 a determination is made as to whether a resource mayrespond to a request (determine whether auto guide is applicable to therequest received). If in operation 1125 it is determined that a resourcemay respond to a request, control is passed to operation 1130 andprocess 1100 continues. If in operation 1125 it is determined that aresource may not respond to a request, control is passed to operation1135 and process 1100 continues. For example, if a guide has providedneeded information for a resource to respond to a request, it may bedetermined that a resource may respond to a request.

In operation 1130 a response to a request is provided by a resource (ananswer is provided automatically). For example, a database, news feed,RSS feed, a translation service, a music recognition service, and/orother system which may provide a response to a user request may provideinformation to a user. Multiple resources may be used to provide aresponse to a request. For example, a music lyrics feed and atranslation service might be used to respond to the query ‘what is thefirst line of ‘Sesame Mucho’ in English?' In at least one embodimentrating information of a result provided may be obtained. Control ispassed to operation 1105 and process 1100 continues.

In operation 1135, information of available answer(s) is evaluated withrespect to information of a query including to determine suitability ofan existing answer to be used to respond to a request. For example, afunction such as the ARP function may be evaluated to determine if aquery may be answered by a stored result. In at least one embodiment,information of a query is provided to an expeditor who may evaluate theability to respond using an existing answer. An expeditor may modify aquery, which may cause a matching function to be reevaluatedautomatically and/or when triggered by an expeditor. Control is passedto operation 1140 and process 1100 continues.

In operation 1140, a determination is made as to whether a stored answermay be used to respond to a request. If in operation 1140 it isdetermined that a stored answer may be used to respond to a request,control is passed to operation 1145 and process 1100 continues. If inoperation 1140 it is determined that a stored answer may not be used torespond to a request, control is passed to operation 1150 and process1100 continues. For example, if an expeditor selects an answerassociated with a previous request, or if an expeditor indicates that ananswer associated with a previous request is not suitable theprobability that a stored answer may be used to respond to a userrequest may be increased or decreased.

In operation 1145 a stored response is provided. For example, an answerstored in the database 120 (FIG. 1) may be provided to a user. Ratinginformation may be obtained based on actions of a user and/or a guide.For example if an expeditor has selected an answer to be provided to auser, a ranking and/or rating of the answer may be increased. Control ispassed to operation 1105 and process 1100 continues.

In operation 1150, information of a category associated with a requestis confirmed. In at least one embodiment, an ambassador may activelyconfirm a category associated with a request. In at least oneembodiment, a category may be not be confirmed when an expeditor doesnot modify the category. Control is passed to operation 1155 and process1100 continues.

In operation 1155 information of a succinct query associated with arequest is confirmed. In at least one embodiment, an ambassador may berequired to confirm a succinct query associated with a request. Controlis passed to operation 1160 and process 1100 continues.

In operation 1160 a request and information associated with the requestis provided to a searcher. For example, a succinct query, a category,user information, search resources, etc., which may be associated with arequest may be provided to a searcher at a searcher system (e.g. thesearcher system 105 (FIG. 1)) using a web server functionality of thesearch system 130 and a web browser functionality of a searcher system.Control is passed to operation 1105 and process 1100 continues.

In operation 1105 information associated with the query vetting process1100 is recorded. For example the database 120 (FIG. 1) may be updatedwith information of a category, a succinct query, a response provided bya resource, a stored response provided, an ambassador, a guide, a query,etc., which have been modified by the process 1100 may be recorded. Inat least one embodiment, usage information of a resource may be recordedto determine compensation for the resource. In at least one embodiment,use of a stored result may cause a rating of the stored result to bereduced. In at least one embodiment an amount of time required by anexpeditor may be recorded in order to evaluate efficiency of theexpeditor. Control is passed to operation 1110 and process 1100continues.

An embodiment of tools for creation of a vetted query is illustrated inFIGS. 12 to 15. The embodiment may be used by an ambassador or expediterto process a request.

As illustrated in FIG. 12, the GUI 1200 includes a customer informationwindow 1202, an advertisement window 1208, a query building window 1210,a category selection window 1230, a location selection window 1238, a‘Send to Guide’ button 1242, a status selection tool 1244, and actioncontrols 1246.

The customer information window 1202 may include a last known locationindicator 1204 and a recent activity indicator 1206. The last knownlocation indicator 1204 may display a user's most recent geographiclocation. This information may be used by the search system 130 and/or aguide to facilitate provision of an answer to a user that is relevant toa certain location, area, region, etc. The recent activity indicator1206 may display a user's most recent requests and/or other contextinformation which may be relevant to a request. For example, the recentactivity indicator 1206 may display a user's previous queries, systemresponses, any advertisements that may have been provided to the user,previous search results, profile information of a user, etc. Theadvertisement window 1208 may display a selected advertisement based oncontent of the GUI 1200, which may be transmitted to a user based onactions of a guide. For example, one or more advertisements which havebeen selected based on content of a user query may be presented in anorder determined by the search system 100 (FIG. 1). An expeditor mayselect an appropriate advertisement or other information based oninformation provided using the GUI 1200, which may be provided to a useras part of a clarification message, during a search by a guide, as aninterstitial message, etc.

Prior to a providing a response to a query, an initial guide orambassador may use the GUI 1200 to develop a well-formed or structuredquery from a user submitted query, categorize the query, provide adatabase response, ask a user for clarification, report abuse to thesearch system, etc., before passing the well-formed query on to anadditional guide, if needed. The query building window 1210 may includequery type controls 1212, a session time indicator 1214, a raw queryindicator 1216, NLP processed query indicator 1248, a query buildingtext box 1220, interrogative word selection controls 1222, a suggestedquestions window 1224, suggested question indicators 1226, and ascrolling control 1228.

The query building window 1210 may be used by a guide to develop awell-formed query based upon a user submitted query. For example, asillustrated in FIG. 12, the query ‘What did Buzz Nutter do?’ has beensubmitted. A guide may use the query type controls 1212 to select a GUIwhich may be used to provide a response to a user request. In at leastone embodiment, the query type controls are provided as text links whichmay cause a GUI such as the GUI's 1400 (FIG. 14) and 1500 (FIG. 15) tobe presented. If a raw query has been determined to match aclassification of query with a pre-determined probability, a queryprocessing GUI may be selected and presented first to an expeditor. Inembodiments, the content of a GUI may be pre-filled with informationbased on a request. For example, if it has been determined that a queryis most likely to be a request for financial information, a GUIassociated with the ‘Stocks’ query type indicator might be providedfirst to an expediter.

Selection of an ‘Expedite’ query type control 1212 a may cause the GUI1200 to be provided. Selection of a ‘Jokes’ query type control 1212 b toprovide the GUI 1400 if the guide determines that a user request may besatisfied by a joke. Selection of a ‘Conversational’ query type control1212 c may result in a GUI for creating a conversational response beingpresented. Selection of a ‘Weather’ query type control 1212 e may resultin the GUI 1500 (FIG. 15) being presented. Likewise a GUI associated a‘Stocks’ query type control 1212 d, and a ‘Directions’ query typecontrol 1212 f may be selected by activating the respective query typecontrols. Any number of query type controls 1212 may be provided in theGUI 1200.The session time indicator 1214 may display an amount of timethat is spent by a guide performing a task using the GUI 1200. Selectionof the query type control may be indicated by underlining, and/or anysuitable indicator such as color, shading, etc., as is well known in theart.

The raw query indicator 1216 may allow a guide to select any element ofa query to include in a well-formed query that may be assembled in thequery building text box 1220. For example, a guide may choose to selectthe named entity ‘Buzz Nutter’ using the raw query indicator 1216. TheNLP processed query indicator 1248 may allow a guide to select anyelement of a succinct query which may have been produced by the searchsystem 130 (FIG. 1) to include in a well-formed query that may beassembled in the query building text box 1220.

The interrogative word selection controls 1222 may include any number ofinterrogative words that a guide my select in assembling a well-formedquery in the query building text box 1220. For example, if a query issubmitted to the search system by a user in a declarative form, a guidemay choose to start a well-formed query by activating the ‘Who’ wordselection control 1222 a, the ‘What’ word selection control 1222 b, etc.

The query building window 1210 may include the suggested questionswindow 1224. The suggested questions window 1224 may include any numberof suggested question indicators 1226 indicating queries which have beendetermined to match an NLP processed query and/or a raw query. Using theexample in FIG. 12 the queries ‘Who was Buzz Aldrin?’, ‘Who was BuzzNutter?’ and ‘Who is Buzz Lightyear?’ are presented in the suggestedqueries window 1224. The scrolling control 1228 may be used to view anyadditional suggested questions that may not be displayed. A guide mayselect a suggested question indicator 1226 by clicking or otherwiseselecting the question. If a guide selects the suggested questionindicator 1226 c, the text of the associated question (i.e. ‘Who wasBuzz Nutter’) is transferred to the query building text box 1220.

The category selection window 1230 may contain the suggested categorieswindow 1232, the categories selection control 1234, and the selectedcategories window 1236. The suggested categories window 1232 may displayany categories determined to be a potential category associated with therequest. For example, as illustrated in FIG. 12, the categories,‘Science>Astronauts’, ‘Sports>NFL’ and ‘Entertainment>Movies’ aredisplayed in the suggested categories window 1232 as a potential ‘match’to the user submitted query, ‘What did Buzz Nutter do?’. A guide may usethe categories selection control 1234 to select any category that maynot be contained in the suggested categories window 1232. If a guideselects a category from the suggested categories window 1232, suchselection may be indicated by the addition of the suggested category tothe selected category window 1234 as shown by the selected categories‘Sports>NFL’ in FIG. 12. In at least one embodiment, only one categorymay be associated with a query at a time.

The location selection window 1238 may include a location selectioncontrol 1240. If a guide determines that a user submitted query may belocation related, a guide may select the location selection control 1240in which case a subsequent guide may better provide an answer to arequest. If the search system 130 (FIG. 1) determines that a user querymay be location related the location selection control may be set toactive by default.

The ‘Send to a Guide’ button 1242 may be selected if a guide hascompleted the assembly, formation, selection or otherwise of awell-formed query, and completed categorization of the query, or otheroperations as may be determined by the search system 130 (FIG. 1). Aguide may select the ‘Send to Guide’ button to pass the vetted query toanother guide, and/or to the search system 130 which may in turn providean answer to a user request based on a vetted query. A guide mayalternatively select from the user controls 1246. The ‘Customer Clarify’user control 1246 a may be selected to obtain clarification of a usersubmitted query and/or obtain additional information from a user thatmay be advantageous in forming a succinct query. The ‘Abort’ usercontrol 1246 b may be used to cancel the current session and close theGUI 1200. The ‘Abuse’ user control 1246 c may be used to report abuse orother prohibited behavior of a user. The status selection tool 1244 maybe used to indicate that a guide desires to change the status of theguide to ‘Away’ and not accept additional queries after completion ofthe current session.

A GUI for selecting an answer 1300 is illustrated in FIG. 13. The GUI1300 may be provided as a ‘pop-up’ if a structured query from thesuggested questions window 1220 (FIG. 12) is selected using thesuggested question indicators 1226. Alternately, the GUI 1300 may beprovided to a guide if it is determined that a structured query matchesa user query, but the probability of a match is below a pre-determinedthreshold. For example, the search system 130 (FIG. 1) may determinethat human judgment is required to confirm that a succinct query indeedmatches a user request. Alternately, if a succinct query has beenselected, and several answers with similar quality ratings areassociated with the succinct query, an expeditor may select an answer.Such information may be used to modify a rating of the answersassociated with the succinct query. In at least one embodiment, if anexpeditor has selected the ‘send to a guide’ control 1242 and the vettedquery is a match to a query indicated in a database, the GUI 1300 may beprovided.

The GUI 1300 includes a user query indicator 1302, a succinct queryindicator 1312, an answer selection window 1304, answer selectioncontrols 1306, an answer rejection control 1308, and a completioncontrol 1310.

The user query indicator 1302 provides information of a user query. Thesuccinct query indicator 1312 provides information of a succinct queryassociated with a user request. The answer selection controls 1306 maybe used to select an answer to be provided responsive to a user request.The answer selection control 1306 a selects the answer ‘Buzz Nutter wasa football player. He was an All-Pro for the Colts in the 1950's.’ Theanswer selection control 1306 b selects the answer ‘Buzz Nutter playedcenter and linebacker.’ The answer selection control 1306 c selects theanswer ‘Buzz Nutter is a Colts All-Pro.’ The answer selection controls1306 may be mutually exclusive radio buttons, or other controls wellknown in the art. The answer rejection control 1308 may be used toindicate that a guide is unsure if the answers are relevant, if none ofthe answers are suitable, that a vetted query may be incorrect, or thata query should be sent to a guide for any reason. The completion control1310 may be used to indicate completion of the selections in the GUI1300.

As illustrated in FIG. 14, the GUI 1400 includes a customer informationwindow 1402, an advertisement window 1408, an answer building window1410, query type controls 1412, a session time indicator 1414, a userquery indicator 1416, a suggested query indicator 1430, response typeselection tools 1418, a provided answer display window 1420, an answerbuilding text box 1422, a ‘Send Answer’ button 1424, a status selectiontool 1426, and user controls 1428.

The customer information window 1402 may include the last known locationindicator 1404 and the recent activity indicator 1406. The last knownlocation indicator 1404 may display a user's most recent geographiclocation. This information may be used by the search system 130 (FIG. 1)and/or a guide to facilitate provision of an answer that is relevant toa certain location, area, region, etc. The recent activity indicator1406 may display a user's most recent use of the search system 130(FIG. 1) and/or other context information which may be relevant to arequest. For example, the recent activity indicator 1406 may display auser's previous queries, responses by the search system 130 (FIG. 1),any advertisements that may have been provided to the user, previoussearch results, profile information of a user, etc. The advertisementwindow 1408 may display a selected advertisement based on content of theGUI 1400, which may be transmitted to a user based on actions of aguide. For example, one or more advertisements which have been selectedbased on content of a user query may be presented. An expeditor mayselect an appropriate advertisement and/or other information based oninformation provided using the GUI 1400, which may be provided to a useras part of a clarification message, during a search by a guide, as aninterstitial message, etc.

A guide may select the ‘Jokes’ query type control 1412 b to provide theGUI 1400 if the guide determines that a user request may be satisfied bya joke. Selection of the ‘Conversational’ query type control 1412 c maycause a ‘Conversational’ GUI to be provided. Selection of the ‘Weather’query type control 1412 e may result in the GUI 4600 (FIG. 46) beingpresented. Likewise a GUI associated with ‘Stocks’ 1412 d, and‘Directions’ 1412 f may be selected. Any number of query type controls1412 may be provided in the GUI 1400. In at least one embodiment a firstGUI provided to an expeditor may be determined based on analysis of aquery.

The answer building window 1410 may include the user query indicator1416, and the suggested query indicator 1430. The joke type selectiontools 1418 may be selected by a guide based on a guide's interpretationof a query in the user query indicator 1416, and/or the suggested queryindicator 1430. For example, as illustrated in FIG. 14, the usersubmitted query, ‘tell me a funny’ may be interpreted by automatedprocessing of a request to be a request for a joke as indicated by thesuggested query ‘What is a funny joke?’ In such a case, the GUI 1400 maybe provided initially to an expeditor instead of the GUI 1200. A guidemay select a category of response using the joke type selection tools1418. For example, the joke category ‘Lawyer’ has been selected asindicated by the response type selection control 1418 e. Alternatively,a guide might select ‘Blonde’ jokes 1418 a, ‘Caveman’ 1418 b, ‘Oneliners’ 1418 c and ‘Religion’ 1418 d. Any number of joke type selectiontools 1418 may be provided.

If a guide selects any of the joke selection tools 1418, responsesassociated with the selected category in for example the search systemdatabase 120 (FIG. 1) may be displayed in the provided answer displaywindow 1420. A guide may select any response displayed in the providedanswer display window 1420 to submit as a response to a query.Alternatively, a guide may use any part of a provided answer and/or theguide's text as a response to a user submitted query by entering suchtext into the answer building text box 1422. In at least one embodiment,a guide may not be allowed to enter free form text in the answerbuilding text box 1422. The text remaining indicator 1423 may displaythe number of characters a guide has remaining to provide an answer inthe answer building text box 1422 based on a total number of charactersand/or words that a guide is allowed to use in building an answer. Thenumber of characters may be based on a user device constraint, or anysystem constraints.

The ‘Send Answer’ button 1424 may be selected if a guide has completedthe assembly, formation, selection or otherwise an answer in response toa query. A guide may alternatively select from the user controls 1428.The ‘Customer Clarify’ user control 1428 a may be selected to obtainclarification of a user submitted query and/or obtain additionalinformation from a user that may be advantageous in forming an answer.The ‘Abort’ user control 1428 b may be used to cancel the currentsession and close the GUI 1400. The ‘Abuse’ user control 1428 c may beused to report abuse or other prohibited behavior of a user. The statusselection tool 1426 may be used to indicate that a guide desires tochange the status of the guide to ‘Away’ and not accept additionalqueries after completion of the current session.14

As illustrated in FIG. 15, the GUI 1500 includes a customer informationwindow 1502, an advertisement window 1508, an answer building window1510, query type controls 1512, a session time indicator 1514, a usersubmitted query indicator 1516, a suggested query indicator 1530,response type selection tools 1518, a provided answer display window1520, an answer building text box 1522, a ‘Send Answer’ button 1524, astatus selection tool 1526, and user controls 1528.

The customer information window 1502 may include the last known locationindicator 1504 and the recent activity indicator 1506. The last knownlocation indicator 1504 may display a user's most recent geographiclocation. This information may be used by the search system and/or aguide to facilitate provision of a query answer to a user that isrelevant to a certain location, area, region, etc. The recent activityindicator 1506 may display a user's most recent use of the search system130 (FIG. 1) and/or other context information which may be relevant to arequest. For example, the recent activity indicator 1506 may display auser's previous queries, responses by the search system, anyadvertisements that may have been provided to the user, previous searchresults, profile information of a user, etc. The advertisement window1508 may display a selected advertisement based on content of the GUI1500, which may be transmitted to a user based on actions of a guide.For example, one or more advertisements which have been selected basedon content of a user query may be presented in an order determined bythe search system 130 (FIG. 1). An expeditor may select an appropriateadvertisement and/or other information based on information providedusing the GUI 1500, which may be provided to a user as part of aclarification message, during a search by a guide, as an interstitialmessage, etc.

A guide may select the ‘Jokes’ query type control 1512 b to provide theGUI 1400 if the guide determines that a user request may be satisfied bya joke. Selection of a ‘Conversational’ query type control 1512 c mayresult in a GUI for creating a conversational response being presented15. Selection of the ‘Weather’ query type control 1512 e may result inthe GUI 1500 (FIG. 15) being presented. Likewise a GUI associated with‘Stocks’ 1512 d, and ‘Directions’ 1512 f may be selected. Any number ofquery type controls 1512 may be provided in the GUI 1500. In at leastone embodiment a first GUI provided to an expeditor may be determinedbased on analysis of the user query.

The answer building window 1510 displays the user query indicator 1516,and the suggested query 1530. The weather type selection tools 1518 maybe selected by a guide based on a guide's interpretation of a query inthe user query indicator 1516, and/or the suggested query indicator1530. For example, as illustrated in FIG. 15, the user submitted query,‘will it be nice today?’ may be interpreted to be a request for aweather information as indicated by the suggested query ‘What is weatherfor Miami, Fla.?’, which is at least in part determined by the locationinformation indicated in the customer information window 1502. In such acase, the GUI 1500 may be provided to an expeditor first instead of theGUI 1200 (FIG. 12). A guide may select a category of response using theweather type selection tools 1518. For example, the weather category‘Current’ has been selected as indicated by the weather type selectioncontrol 1518 a. Alternatively, a guide might select the ‘1-Day’ 1518 b,‘Tomorrow’ 1518 c, ‘Week’ 1518 d and ‘Extended’ 1518 e weather typeselection tools. Any number of weather type selection tools 1518 may beprovided.

If a guide selects any of the weather type selection tools 1518,responses associated with the selected type may be displayed in theprovided answer display window 1520. A guide may select any responsedisplayed in the provided answer display window 1520 to submit as aresponse to a query. Alternatively, a guide may use any part of aprovided answer and/or the guide's text as a response to a usersubmitted query by entering such text into the answer building text box1522. In at least one embodiment, a guide may not be allowed to enterfree form text in the answer building text box 1522. The text remainingindicator 1523 may display the number of characters a guide hasremaining to provide in the answer building text box 1522 based on atotal number of characters and/or words that a guide is allowed to usein building an answer. The number of characters may be based on a userdevice constraint, or any system constraints. In the case of a requestwhich may be processed using an automated resource, an answer may beprovided without requiring selections by a guide in order to optimizethe time required to respond.

The ‘Send Answer’ button 1524 may be selected if a guide has completedthe assembly, formation, selection or otherwise an answer in response tothe user submitted query. A guide may alternatively select from the usercontrols 1528. The ‘Customer Clarify’ user control 1528 a may beselected to obtain clarification of a user submitted query and/or obtainadditional information from a user that may be advantageous in formingan answer. The ‘Abort’ user control 1528 b may be used to cancel thecurrent session and close the GUI 1500. The ‘Abuse’ user control 1528 cmay be used to report abuse or other prohibited behavior of a user. Thestatus selection tool 1526 may be used to indicate that a guide desiresto change the status of the guide to ‘Away’ and not accept additionalqueries after completion of the current session.

Efficient utilization of human assistants or guides in a human assistedsearch system is an important element of success. Better utilization ofthe expeditor and searcher is desirable, but may be elusive. The systemand method disclosed herein combines the well proven expeditor conceptwith the use of Natural Language Processing NLP to further enhancesystem efficiency. An NLP process is implemented to ensure that arequest is processed using machine based processes which may improve thequality of the content of a request. NLP is used to convert a userrequest into a suggested structured query which may be compared to adatabase of queries.

NLP is used to create estimation and matching functions which may beused to determine the probability that a response to a request isavailable. NLP is further used to automatically categorize a query tomatch with a system index. Results of NLP are used to determine whethera query may be answered automatically, may be routed to an expeditor, ormay be routed to a searcher.

NLP processing is implemented in a multi-layered approach. In a firstlayer, an unprocessed request or ‘raw query’ is passed through a seriesof automated processes which may improve the readability of a query,correct spelling, tokenize the query, detect named entities, determinelocations, produce an annotation graph, etc., prior to the query beingcompared to a database of queries.

A target query is compared to a database of reference queries todetermine whether a match to the query is found. If a reference querywhich is an exact match to a target query is found, a quality functionassociated with any answers associated with the matching query isevaluated, and is used to determine whether the answer should beautomatically sent, and/or should be presented to a guide (e.g. andexpediter) to determine whether the answer should be sent responsive tothe target query.

If a reference query is not found which exactly matches a target query,reference queries which are highest ranked based on an index match tothe target query may be evaluated to determine if the queries are asufficiently close match to a target query. In at least one embodiment,reference queries which have similar scores from the index may becompared to each other to determine whether any answers associated withthe reference queries may be evaluated if a reference query isdetermined to be a suitable match to a target query. This may allow ahigher quality answer associated with a lower ranked reference query tobe provided responsive to a request including the target query.

Automated analysis of a query may be used to determine whether a requestis to be presented to an expediter guide, and/or may be presented to asearcher guide directly. Information such as query length, querycategory, keywords of a query, etc., may be used to determine whether arequest is provided to an expediter, and/or may determine a toolsetand/or an order of presentation of a toolset to a guide.

If a request is presented to an expediter, an expediter may modify aquery associated with the request and the modified query may be used asa target query for NLP analysis of the request. An expediter may bepresented with results of comparisons to a revised target query, whichmay allow an expeditor to select and/or review a matching query and/oranswer. NLP processing may be adjusted based on whether a guide hasreviewed a request.

If it is determined that a query is to be routed to an expeditor, a GUIpresented to an expeditor may be based at least in part on evaluation ofthe type of response which an expeditor may need to provide to thequery. This may significantly improve efficiency of the expeditorprocess resulting in better user experience. An expeditor may select anadvertisement to be provided to a user, which may significantly improvetargeting of an advertisement, which may be highly valuable as anadvertiser may obtain the exclusive attention of a user. Further a welltargeted advertisement may be provided while a user is waiting for asearch result.

The system described may increase fully automated responses as a userquery may be standardized using algorithmic preprocessing which mayincrease matching to a database of reference queries. An expediter isnot required to correct common errors in punctuation, etc. which may beeasily corrected automatically. An expediter may modify a query toconform to a standardized query or otherwise, which may improve a matchto a reference query. If a target query is matched by a guide to areference query, information of the matching may cause the target queryto be used as a reference query subsequently.

Use of NLP algorithms has been demonstrated to greatly improve automatedand human-assisted matching of a user request to answers. In the case ofthe ChaCha® answer service, use of blended processing including NLPprocessing has reduced the manpower requirements to respond to a queryby an order of magnitude compared to processing by humans alone, whilemaintaining the same level of answer quality. For example, the medianlength of queries answered automatically using the new system may beequal to the median length of queries answered by human expediters givenan equivalent size corpus of reference queries. Further, a percentage ofqueries requiring a search may be reduced by as much as eighty percentusing the methods and systems described herein.

While the embodiments are described in terms of a generalized answerservice, the methods and system are not so limited. The methods andsystems might be implemented in a customer service operation, aninternal enterprise information network, and/or any system whichresponds to user requests for information.

Any or all of the operations described herein may be implemented via oneor more hardware components. However, the present invention is notlimited to any specific implementation of an operation. For example, oneor more operations discussed herein may be implemented via softwareexecuted on a device while others may be executed via a specifichardware device.

The present invention may be implemented using a program stored, forexample, in a computer-readable storage medium such as a CD-ROM, etc.,or using one or more specialized terminals, devices or systems that isenabled to execute operation(s) described herein. The storage orrecording medium used in an embodiment can be selected from amongvarious computer-readable media including, a disk, a DVD, an internalstorage device (memory such as RAM or ROM) in a computer, etc.

As mentioned above, the embodiments can be implemented in computinghardware (computing apparatus) and/or software, such as (in anon-limiting example) any computer that can store, retrieve, processand/or output data and/or communicate with other computers. The resultsproduced can be displayed on a display of the computing hardware. Aprogram/software implementing the embodiments may be recorded oncomputer-readable media comprising computer-readable recording media.The program/software implementing the embodiments may also betransmitted over transmission communication media. Examples of thecomputer-readable recording media include a magnetic recordingapparatus, an optical disk, a magneto-optical disk, and/or asemiconductor memory (for example, RAM, ROM, etc.). Examples of themagnetic recording apparatus include a hard disk device (HDD), aflexible disk (FD), and a magnetic tape (MT). Examples of the opticaldisk include a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM(Compact Disc-Read Only Memory), and a CD-R (Recordable)/RW. An exampleof communication media includes a carrier-wave signal.

Further, according to an aspect of the embodiments, any combinations ofthe described features, functions and/or operations can be provided.

The many features and advantages of the claimed invention are apparentfrom the detailed specification and, thus, it is intended by theappended claims to cover all such features and advantages of the claimedinvention that fall within the true spirit and scope of the invention.Further, since numerous modifications and changes will readily occur tothose skilled in the art, it is not desired to limit the invention tothe exact construction and operation illustrated and described for thedisclosed embodiments, and accordingly all suitable modifications andequivalents may be resorted to, falling within the scope of the claimedinvention. It will further be understood that the phrase “at least oneof A, B and C” may be used herein as an alternative expression thatmeans “one or more of A, B and C.”

1. A computer-implemented method, comprising: modifying an originalquery received based on a processing applied to the query prior toperforming a search; and performing the search using the modified queryand providing a response to the query as a result of the search.
 2. Thecomputer-implemented method of claim 1, wherein the processing is anatural language processing.
 3. The computer-implemented method of claim1, comprising: determining a process to be carried out for responding tothe original query based on said modifying.
 4. The computer-implementedmethod of claim 1, wherein said modifying includes mapping the originalquery to a vetted query, and using the vetted query to produce aresponse to the original query.
 5. The computer-implemented method ofclaim 1, wherein said modifying includes augmenting the original querybased on review by a human ambassador.
 6. A computer-implemented method,comprising: receiving a request for information; comparing the requestwith a predefined query pattern; correcting spelling; substitutingwords; removing signatures using information of stored requests;creating automatically an annotation graph for the request; selecting acategory; obtaining queries determined to match the annotation graphusing an index of queries; calculating a matching function comparing thequeries to the annotation graph; calculating a rating of a search resultassociated with the queries; ranking the queries based on the matchingfunction; ranking the search results based on the rating; and providinga highest ranked search result responsive to the request.
 7. Thecomputer-implemented method of claim 6, wherein the correcting, thesubstituting, the query pattern, the removing and the creating aremodified based on a source of the request.
 8. The computer-implementedmethod of claim 6, wherein the queries conform to a predeterminedgrammar construct.
 9. The computer-implemented method of claim 6,comprising: providing the request and an output of the comparing,correcting, substituting, removing, and creating to a human searcher;providing to the human searcher a toolset selected based on the output;and selecting by the human searcher a method for responding to therequest.
 10. The computer-implemented method of claim 9, comprising:constructing by the human searcher a sentence; and calculating thematching function and the rating using the sentence.
 11. Thecomputer-implemented method of claim 10, comprising: providing a list ofsearch results in order of the ranking to the human searcher; the humansearcher selecting a search result; and ranking the result selected bythe human searcher as the highest ranked result.
 12. Thecomputer-implemented method of claim 6, wherein the matching functioncomprises: comparing a noun phrase of the stored query to a noun phraseof the annotation graph; comparing a noun of the stored query to a nounof the annotation graph; comparing a named entity of the stored query toa named entity of the annotation graph; comparing a location of thestored query to a location associated with the request; and comparing acategory of the stored query to a category of the request.
 13. Thecomputer-implemented method of claim 6, wherein the rating comprises:comparing a time stamp of an answer to a time stamp of other answers;comparing a usage frequency of an answer to a usage frequency of otheranswers; comparing a logic flag of an answer to a logic flag of otheranswers; comparing a category of an answer to a category of otheranswers; and combining the comparisons arithmetically.
 14. A system,comprising: a search service device receiving a search query, performinga natural language processing of the search query, comparing the searchquery to a corpus of requests, and determining a process for respondingto the search query; a searcher device receiving information of thesearch query, and providing a response to the search query; and adatabase including information of search queries and search results. 15.The system of claim 14, wherein the determining includes selecting anautomated process, a human assisted process, and a combined humanassisted and automated process; wherein the comparing is based on wordfrequency; and wherein the natural language processing is adapted toinformation of a device submitting the request.
 16. A computer readablestorage medium storing therein a program for causing a computer toexecute an operation including responding to an inquiry, comprising:receiving the inquiry; processing the inquiry based on semantic rules;ranking interrogatives using a result of processing of the inquiry;determining a procedure; and initiating the procedure.
 17. The computerreadable storage medium of claim 16 wherein the operation comprises:comparing the inquiry to a template associated with an automatedprocess; substituting a word indicated in the inquiry with a synonym ofthe word; replacing a word indicated in the inquiry with a highestranked word from a lexicon; selecting a lexicon for the processing;removing leading and trailing text from the inquiry which matches aprevious inquiry; and associating a category with the inquiry using wordfrequency associated with an index.
 18. The computer readable storagemedium of claim 16 wherein the operation comprises: receiving a firstmessage from a first source; receiving a second message from a secondsource; and determining the inquiry based on the first message and thesecond message.
 19. The computer readable storage medium of claim 16wherein the operation comprises: ranking interrogatives using theinquiry.
 20. The computer readable storage medium of claim 16 whereinthe operation comprises: determining a probability that the procedure isan automated procedure; presenting the inquiry and the result to a firsthuman guide; providing a suggested response to the first human guide;and performing a procedure indicated by the first human guide.
 21. Thecomputer readable storage medium of claim 20 wherein the operationcomprises: transferring by the first human guide the inquiry to a secondhuman guide; and adjusting a parameter used to determine the probabilitybased on the procedure indicated by the first human guide.
 22. Thecomputer readable storage medium of claim 20 wherein the operationcomprises: modifying the inquiry by the first human guide; adjusting aparameter of the ranking; and ranking the interrogatives based on themodified inquiry.
 23. The computer readable storage medium of claim 16,wherein the operation comprises: determining the procedureautomatically; and initiating the procedure by a human guide.
 24. Thecomputer readable storage medium of claim 16, wherein the operationcomprises: determining a probability that an automated response may beprovided; and selecting a human guide to respond to the inquiry based onthe probability.
 25. The computer readable storage medium of claim 16,wherein the operation comprises: receiving a first message from a firstsource; receiving a second message from a second source; determining theinquiry based on the first message and the second message; comparing theinquiry to a template associated with an automated process; substitutinga word indicated in the inquiry with a synonym of the word; replacing aword indicated in the inquiry with a highest ranked word from a lexicon;selecting a lexicon for the processing; removing leading and trailingtext from the inquiry which matches a previous inquiry; and associatinga category with the inquiry using word frequency associated with anindex; ranking interrogatives using the inquiry; determining aprobability that an automated response may be provided; selecting ahuman guide to respond to the inquiry based on the probability;presenting the inquiry and the result to a first human guide; providinga suggested response to the first human guide; performing a procedureindicated by the first human guide; transferring by the first humanguide the inquiry to a second human guide; adjusting a parameter used todetermine the probability based on the procedure indicated by the firsthuman guide modifying the inquiry by the first human guide; ranking theinterrogatives based on the modified inquiry; ranking a plurality ofresponses associated with the interrogatives based on a selection by athird human guide, temporal information, and location information;receiving a selection by the first human guide; and modifying a rankingof the plurality of responses based on the selection.